192 ответов в этой теме

[0x30A88]

Mephales, give the site some time. That latency have happened to me a couple of times and I did just what you did the first time I experienced said lag.

[Mephales]

Thought Process wrote...

I don't know anything about python but you should profile that code to see what the bottlenecks are, resaving a large GFF file seems to take way longer than it should.

I'll do that. The entire reason there's a lazy reader is because opening GFFs is slow as well. Hopefully it's something simple like using unbuffered I/O... Hmm, I wonder if...

[Mephales]

Oh bugger, found another bug. The Save As YAML option actually took over the normal Save As as well. I'll have a fixed version up within the hour, along with some faster saving (I haven't profiled it yet, but to take out the file I/O entirely I am now building the serialized GFF in memory first then writing it to disk.)

[Mephales]

OK, 0.4.5c is up. It has a fixed Save As option, which if 0.4.5b is the version you were trying to save in Though Process, it wasn't even saving the right data. I also do a lot more bulk I/O, so the time is takes to save and load is mostly my code's fault now. I'll get to profiling it soon.

[Thought Process]

No rush, I was only commenting on my observations (was using your tool to load/save to test my GFF serializing/deserializing code).

[Mephales]

Maybe not such a good idea... Something seems off with my loading or saving code. I'm debugging it now.

[Mephales]

Except now I can reproduce the area transition bug without using any edited save games at all, and with all recently installed mods uninstalled. So I guess my editor isn't to blame. Good for everyone else, not so much for me.

[daywalker03]

Damn; I'm trying to figure out the alterations to the msh format and it's driving me slightly buggy.

[Eshme]

Mephales wrote...

What do you mean by a live example, and what would you be using it for? Dragon Age 2 can handle normal filenames, the new ERF format just makes them optional by adding a hash of the filename that it uses for faster lookup.

I mean i am lookign for a example to convert filenames into a hash. It seems i found it in your scripts i guess, i only need to make it work in maxscript instead. It dont support 64 bit datatype, and no overflow really. Its tricky i need to have an exact example to reduce time i am wasting due limits in maxscript. If i can even manage to i dont know.

But then again i am uncertain if a unzipping algorithm can be made in maxscript. Since the resources are compressed.

I am developing 3dsmax importer and exporter scripts, just so u know.

[tmp7704]

Eshme wrote...

I mean i am lookign for a example to convert filenames into a hash.

http://www.isthe.com.../fnv/index.html

has all details and some source code, hopefully it can be of some help.

[tmp7704]

daywalker03 wrote...

Damn; I'm trying to figure out the alterations to the msh format and it's driving me slightly buggy.

Is there any actual alterations? Tried to open a few meshes with DAO importer but didn't spot any issues. Maybe just didn't run into the right ones... the headers indicated .msh are "MESH V1.0 PC" i.e. same as DAO.

Сообщение изменено: tmp7704, 05 Март 2011 - 09:41 .

[daywalker03]

Odd; DragonBlender isn't liking them for some reason.

[Eshme]

tmp7704 wrote...

Eshme wrote...

I mean i am lookign for a example to convert filenames into a hash.

http://www.isthe.com.../fnv/index.html

has all details and some source code, hopefully it can be of some help.

Thanks. And what Prime do Bioware used? I mean exactly for DA2. And which offset basis?

[Eshme]

tmp7704 wrote...

daywalker03 wrote...

Damn; I'm trying to figure out the alterations to the msh format and it's driving me slightly buggy.

Is there any actual alterations? Tried to open a few meshes with DAO importer but didn't spot any issues. Maybe just didn't run into the right ones... the headers indicated .msh are "MESH V1.0 PC" i.e. same as DAO.

Its the same format. Although some new datatypes are in use (normalized bytes for example). My importer chokes too

[Thought Process]

Eshme wrote...

Thanks. And what Prime do Bioware used? I mean exactly for DA2. And which offset basis?

Same as on that page.

[daywalker03]

Eshme wrote...

tmp7704 wrote...

daywalker03 wrote...

Damn; I'm trying to figure out the alterations to the msh format and it's driving me slightly buggy.

Is there any actual alterations? Tried to open a few meshes with DAO importer but didn't spot any issues. Maybe just didn't run into the right ones... the headers indicated .msh are "MESH V1.0 PC" i.e. same as DAO.

Its the same format. Although some new datatypes are in use (normalized bytes for example). My importer chokes too

Nomalized bytes are listed in the constants, but that doesn't seem to be the problem. I may have to try some things to see what's going on with this.

[tmp7704]

daywalker03 wrote...

Nomalized bytes are listed in the constants, but that doesn't seem to be the problem. I may have to try some things to see what's going on with this.

Some models use weird streams for some components, so maybe that's something to watch for. Like 4 half-floats for definition of single UV channel (the relevant data seems to be carried by first two of the half-floats. Maybe the other two carry some extra info but i haven't checked)

[Eshme]

Any idea why exporting is so slow here? I think it was faster before, but then i havent exported a big file. It ran an hour and only 14MB exported yet. Like a handful of files.

Oh yea version4.2 is over 1000 times faster.

And something strange about the progress bar too, it goes backwards sometimes lol

Thought Process wrote...

Eshme wrote...

Thanks. And what Prime do Bioware used? I mean exactly for DA2. And which offset basis?

Same as on that page.

And thanks , its a big help. I thought i saw multple variants of the hashing but it seems there is only 1 variant. Maxscript cannot do unsigned integer operations and acount for overflow. Not to meantion 64 bit. Them values are out of range. I need bitoperations lol! So it helps starting out with proven values and algorithms.

Сообщение изменено: Eshme, 06 Март 2011 - 07:16 .

[Mephales]

0.4.5d is uploaded.

Eshme wrote...

Any idea why exporting is so slow here? I think it was faster before, but then i havent exported a big file. It ran an hour and only 14MB exported yet. Like a handful of files.

Oh yea version4.2 is over 1000 times faster.

Yeah, I stopped reading the entire compressed data into memory, but my replacement only read a byte at a time... I added a sizehint parameter that lets the decompressing class read larger chunks of data blah blah blah.

And something strange about the progress bar too, it goes backwards sometimes lol

Yeah, it's not a progress bar, but an activity bar, just to show the user it's alive. I suppose in this case I could make it a progress bar, though...

At any rate, I still have no conclusive proof of whether my editor is screwing up my savegames or not... My current theory is that since Python has only 64-bit floating point, the conversion from float to double and back to float may be mucking up something. I was able to change my code to read floats as ints, resave a savegame in my editor, and play without issue. But if I resave it while it's converting floats to doubles and back on save, eventually I start getting issues where transition points don't work, chests can't be opened, and conversations can't be started. Weirder still, I can exit to the main menu, load a good save, and the same problems exist. Exit the game and load a good save, no issues... It's like the the floats are being changed to a slightly different value that results in whatever code handles triggers being thrown into a non-working state.

At any rate, the upshot is that I'll finally start work on writing a new class for reading and writing GFF files, one that acts as a view of the binary data, not only loading values lazily like my LazyGFF class, but also writing changes immediately to the backing data. This means that a save will just involve dumping the binary data to disk instead of generating the binary data from the Python structures like it currently does. It'll also means that the entire file will not have to be parsed like it currently has to be, nor will any of those unedited values be changed.

[Thought Process]

If you're worried, internally store floats as their int value (read it from the file as an int), so you safely can rewrite the value back with no issues. Then when it comes to display it, convert it on the spot, same for editing. Maybe storing the four bytes rather than an int would be more efficient? I don't know python.

Сообщение изменено: Thought Process, 06 Март 2011 - 06:11 .

[Eshme]

I am sorry to disturb the geekdom of this thread, may i ask for a programming favor? In face of the compression, i cannot use the zlib libraries with maxscript. And cant even get binary written into files.

Would the be the possibility of a commandline tool to either take the ERF, outputpath and the streamoffset/bytelength as arguments and dump it out to a file with first "F9" replaced by "78 9C"? Or even write it out and unzip it? Or eventually take just the ERF and the filename to unzip and the target, but that is not really needed since i can find the offset. Or something?

And if possible also a commandline tool to convert a text file with all hex strings convert into binary? The text file consist of only text like "ff99aacc" etc and newline chars. hee

But i have found a tool to unzip, it just needs the first byte replaced. (gives me an buffer error yet still works it seems) So the unzip is not entirely needed.

If anyone has the time only. I would rather have it like before, that every filename is stored in plaintext rather than a hash. That makes stuff abit tough to find for modders, unless it is within a linked filestructure what a full scale importer can manage.

[Thought Process]

Making an ERF decompression tool should be pretty simple. Doing ERF 3.0 -> ERF 2.2/2.0 would be lossy if any hashes are unknown.

[Mephales]

I can certainly add an additional executable to my editor that's a standalone ERF extractor. How do you want the command line to be? erfextractor [--password <erf password>] <erf file> <resource to extract> <destination>? Do you want to specify the directory to extract to, or the full path including filename, so you can name it whatever you like. If instead you want to specify the directory, and the resource has directories in its name (like the art/* files), do you want all the directories created, or do you want the file created in the exact directory you specify? How about errors?

[tazpn]

I have a full ERF extraction tool that unpacks about 14% of the filenames. Unfortunately, the remainder of names have to have smarter algorithms applied to work be guessed. I suppose the hash list could just be used by this tool if it desired. I suppose I might publish my stuff though I'm not planning to support it and it would probably be better here.

Edit:  I've uploaded my tools here.   Please feel free to use the hashes that I've already deduced.  The Resources\\da2.hash file is for all files.  The Resources\\ext.hash file is for file extensions used in ERF files.  I did some minor updates to my DAO command tools to verify things work. The DDS files are largely missing.  I think because most filenames are relative.  It will automatically add the ERF file name and relative path to the output path.

Usage: da2cmd erf "C:\\Games\\RPG\\Dragon Age 2\\*" -d c:\\temp\\files

Сообщение изменено: tazpn, 07 Март 2011 - 06:54 .

[Thought Process]

tazpn wrote...

I have a full ERF extraction tool that unpacks about 14% of the filenames. Unfortunately, the remainder of names have to have smarter algorithms applied to work be guessed. I suppose the hash list could just be used by this tool if it desired. I suppose I might publish my stuff though I'm not planning to support it and it would probably be better here.

What ERF(s) did you encounter type 0x8DD70FE1 in?