192 respuestas en este tema

[Thought Process]

It's encoded in some bizaare way that I can only assume they created that scheme to obfuscate the string table.

Edit: it also functions as a form of compression so it's not a total butchery of things, though I don't see why they couldn't have just rolled a scheme that used a standard compression method.

Editado por Thought Process, 03 marzo 2011 - 11:02 .

[Thought Process]

I wrote a tool which can successfully convert TLK V0.5 to TLK V0.2. Will probably commit to my repository later.

[Eshme]

Weird so the filenames are hashes, that cannot be decoded. How does the game see the names then? Cannot there be a brute force hash lockup for us?
Nice progress you guys made btw.

[Thought Process]

Bruteforcing isn't really feasible given that most file names are 8+ characters in length, though it's better than usual since we know what the extension is supposed to be, smaller file names should bruteforce easily.

DA2 hashes file names at runtime then uses those hashes to look up file data. It technically doesn't need to have file names if everything was moved over to store a hash instead of a file name.

Editado por Thought Process, 03 marzo 2011 - 06:40 .

[Eshme]

ooh yea right. For a file reference, you will have to make a hash and lock it up instead. Do you have a live example to make one of the hash 32 and 64 bit for me please? I have some problems doing that in maxscript alone lol. Dataformats there are not suited for this.

[Mephales]

What do you mean by a live example, and what would you be using it for? Dragon Age 2 can handle normal filenames, the new ERF format just makes them optional by adding a hash of the filename that it uses for faster lookup.

[tmp7704]

Hmm if the new version can use hashed filename as reference to a file and the actual name doesn't need to be known at this point, i wonder what's the procedure for file override.

Editado por tmp7704, 03 marzo 2011 - 11:43 .

[Thought Process]

Same as it ever was, like I said: Dragon Age 2 will hash file names that it loads by name so they can be looked up in that manner.

[Mephales]

And 0.4.4 is up. The most obvious change is there's now a V3.2 viewer as well. I might make it an editor in the future, but I figured with so many V3.2 files in the game I ought to put at least that much in. I also modified the erf module a bit, but the editor doesn't really take advantage of the improvements.

[tmp7704]

Thought Process wrote...

Same as it ever was, like I said: Dragon Age 2 will hash file names that it loads by name so they can be looked up in that manner.

But if the only thing you have to work with as file id is the hash of original name/path, then how do you know what name should be given to the file in the override folder, so that the hash matches and the file actually overrides the original?

For some file types that name can be looked up in the GFF structure, but there's files which don't carry such info. Texture files being one example.

Editado por tmp7704, 04 marzo 2011 - 02:44 .

[Thought Process]

You find the names in other locations, the game obviously has to reference them somehow. If the game only references the hash somewhere, then you're SOL unless you manage to bruteforce it.

[daywalker03]

File names, minus extensions, are in the gda files; in fact, I believe even the specific virtual directory is identified.

[tmp7704]

Mephales wrote...

And 0.4.4 is up. The most obvious change is there's now a V3.2 viewer as well. I might make it an editor in the future, but I figured with so many V3.2 files in the game I ought to put at least that much in. I also modified the erf module a bit, but the editor doesn't really take advantage of the improvements.

Seems to be a bug with 0.4.4 -- opening the artfp.erf from DA2 demo only shows a handful of files as content (even though "export all" will over course of few hours export more than 4000 files from the archive) The previous version didn't have that --or at least listed more files than the new one-- so that appears to be new thing.

[Mephales]

Depending on what you mean, it's either a bug I can't replicate or something I did intentionally. Do you mean the list of names is shorter, or that there are the same number of filenames but fewer are the real filename? I did delete all the filenames from the packaged cache, leaving only the file extensions. Somewhat futile, as I'll have to find a new host in a few versions just the same, but it did save some space. I can upload my current cache as a separate download. This assumes that this is the problem. When I open artfp.erf I get thousands of filenames, mostly the made up z_*.* ones.

[Thought Process]

Successfully bruteforced all of the column names for worldvault.gda.

(this file is used by Dragon Age 2 to determine what data is imported from a Dragon Age: Origins save file)

Editado por Thought Process, 05 marzo 2011 - 12:34 .

[tmp7704]

Mephales wrote...

Depending on what you mean, it's either a bug I can't replicate or something I did intentionally. Do you mean the list of names is shorter, or that there are the same number of filenames but fewer are the real filename?

 
I most likely misinterpreted result of the changes you did. When i choose to open "artfp.erf" with v.0.4.4 the requester which allows you to pick the exact file to open lists only files with z_* names. With v.0.4.3 somes of the files listed show their directory and file name instead.  I presumed from that the list of files in 0.4.4 contained less files. If instead you just got rid of the path/filenames and switched to showing just the hashes ... well, the 0.4.3 approach made it considerably easier to quickly find and/or figure out the purpose of the files, without it that often becomes rather unfeasible as looking for some specific asset may require just opening them one by one to check for their filenames stored in GFF structure and such.

[daywalker03]

Thought Process wrote...

Successfully bruteforced all of the column names for worldvault.gda.

(this file is used by Dragon Age 2 to determine what data is imported from a Dragon Age: Origins save file)

Nice one.

[Mephales]

tmp7704 wrote...
I most likely misinterpreted result of the changes you did. When i choose to open "artfp.erf" with v.0.4.4 the requester which allows you to pick the exact file to open lists only files with z_* names. With v.0.4.3 somes of the files listed show their directory and file name instead.  I presumed from that the list of files in 0.4.4 contained less files. If instead you just got rid of the path/filenames and switched to showing just the hashes ... well, the 0.4.3 approach made it considerably easier to quickly find and/or figure out the purpose of the files, without it that often becomes rather unfeasible as looking for some specific asset may require just opening them one by one to check for their filenames stored in GFF structure and such.

I understand the problem, and I was probably premature in removing the database before completing more of what I'd planned. I've uploaded 0.4.5. There are now two options under settings for importing filenames and extensions from text files, one filename/extension per line, just like Thought Process's .typelist and .filelist files. In addition, I've uploaded as a separate file a database with a few thousand filenames, most from Thought Process's files, but a few from my own trawling of the 2das in 2da.erf. Just replace the fnvhashes.sqlite in whatever folder you extracted gffeditor-*.7z to with the one in the fnvhashes-*.7z, and you should see some meaningful filenames when you open ERFs again.

Now I really need to play some DA:O, 'cause DA2 comes out Tuesday and I, uh, haven't finished my first playthrough of Origins, much less Awakening or any of the other DLC.

[daywalker03]

Mephales wrote...

Now I really need to play some DA:O, 'cause DA2 comes out Tuesday and I, uh, haven't finished my first playthrough of Origins, much less Awakening or any of the other DLC.

[Thought Process]

Probably off topic, but I found something neat in the demo GUI data files. I had noticed that DA2 is called "dae" (Dragon Age: Origins was "dao"), I wondered what that 'e' stood for, found an answer in the GUI files.



Edit: and with that name, I found this thread.

Editado por Thought Process, 05 marzo 2011 - 03:48 .

[daywalker03]

I'd wondered that too; I do remember seeing that thread, now that you mentioned it.

[tmp7704]

Mephales wrote...

In addition, I've uploaded as a separate file a database with a few thousand filenames, most from Thought Process's files, but a few from my own trawling of the 2das in 2da.erf. Just replace the fnvhashes.sqlite in whatever folder you extracted gffeditor-*.7z to with the one in the fnvhashes-*.7z, and you should see some meaningful filenames when you open ERFs again.

Great, thank you   there seems to be problem with this in v 0.4.5 though -- it doesn't show the expanded filenames no matter if i try the new fnvhashes file or the one from v.0.4.3... build 0.4.4 uses either of these files properly so it appears to be new thing. That's just to let you know for the next update after the DAO playthrough or whenever

[Mephales]

And that's what I get for not testing my changes to the cache. I uploaded 0.4.5b with the fix. It also includes a YAML export, which you can safely ignore, since the editor can't import YAML yet. The exports also come out to be 3-4 times the original size. About their only utility is that you can export two savegames and diff them to identify changes.

Sadly, I lost all progress I made tonight, and possibly further back. Might be a bad mod, might be something else, but I can't leave the camp.

[Mephales]

Argh, lost my previous comment.

Yeah, I got clever with the cache... and forgot to test it. I've uploaded a new version with a working cache. It also has a YAML export, but its only real utility is in exporting two savegames and diffing them to find the changes. The resulting files are huge, and can't be imported back in.

Edit: Or maybe I didn't lose it.... Site seems to be acting odd tonight, I had issues managing the files in my project too.

Editado por Mephales, 05 marzo 2011 - 05:17 .

[Thought Process]

I don't know anything about python but you should profile that code to see what the bottlenecks are, resaving a large GFF file seems to take way longer than it should.