Aller au contenu

Photo

[Release] pyGFF 0.5 ALPHA


  • Veuillez vous connecter pour répondre
192 réponses à ce sujet

#1
Mephales

Mephales
  • Members
  • 83 messages
If you have no idea how to run a Python program from the command line, this release is of limited interest... Unless enough interest is shown to convince me to upload a compiled version somewhere it will fit.

After having essentially abandoned my efforts (and Dragon Age...), I've released a new version of my python package for manipulating GFF files, pyGFF at social.bioware.com/project/1936/. The major improvement is that the viewer is now an editor. It doesn't have all the features of the Toolset's editor: It can't add or remove fields from structures for instance, and you can't define new structures. What you can do is opening an existing file and edit it to your heart's content.

Under the hood, the data structures are a little improved but not where I'd like them, and some of the logic is hackey.

For the future, I plan on adding support for moving structures from one list to another, or replacing a structure with another. Essentially copy-paste. I also plan on writing a new module for reading and writing GFF files on-the-fly, allowing the editor to open files immediately instead of taking upwards of a minute to open a file, and similar gains on saving files... At the expense of responsiveness when you explore new parts of the file. There's a few other improvements I want to make, like additional ways to search an opened file for data.

A quick overwiew of the editor: The editor requires Python 2.6 amd wxPython. Once you get pygff extracted, go to the folder you extracted it to, and run the module gff4.editor (pythonw -m gff4.editor). The File menu has your typical Open/Save/Quit options. When you open a file, the structure shows up in the largest sectionof the editor as a tree, When you select a node in the tree, the bottom pane let you edit it. Save saves the edited value, reset sets it back to whatever is saved. When you have a Create or Delete button, you can add or delete optional values. The list editor should be self-explanatory. Right clicking on a list or structure and choosing Search lets you limit future searches to just that node's children. The search uses regexen at the moment, though strings without characters considered special in regexen will act like your normal substring search. Double-click on results to select that item in the tree.

I'll look into getting a compiled version of the editor uploaded elsewhere... It will not fit here.

Modifié par Mephales, 13 mars 2011 - 08:24 .

  • luna1124 aime ceci

#2
Mephales

Mephales
  • Members
  • 83 messages
Nevermind, 7zipped it fits fine here, though I think keeping historical release is out of the question. It's now the featured file for the project. It's standalone, does not require python to be installed, and ready for anyone who wants to play with it.

#3
Mephales

Mephales
  • Members
  • 83 messages
Quick update, version 0.3 is up, both compiled and source. Files will open much faster now. There is also now cut and paste support in tree view, so you can move or copy pieces of the file around, assuming they fit.You can also paste those pieces into the bottom-right list, the Shelf, then later cut or copy from that list. A few other minor changes were made as well, among them you can now drag-and-drop a file onto the editor and it will open.

In case it wasn't obvious, just extract gff4editor-0.3.7z where you like and double click on editor.exe.

I hope someone finds some use in this... and that Dragon Age 2 uses GFF V4.0 as well.

Modifié par Mephales, 31 octobre 2010 - 11:02 .


#4
Mephales

Mephales
  • Members
  • 83 messages
I've uploaded version 0.4. It appears I only half-lucked out: Dragon Age II does use GFF V4.0, but not ERF V2.x which I had programmed but never uploaded. I have now, along with a fix for circular structure definitions, like RIM structures in CIF files.

As more information on ERF V3.0 is discovered, I'll add support, but all I've got now is the ability to read the table of contents.

As always, I hope someone finds a use for either the editor or the python modules, and I'm always open to criticism and suggestions.

Modifié par Mephales, 26 février 2011 - 07:13 .


#5
daywalker03

daywalker03
  • Members
  • 357 messages

Mephales wrote...

I've uploaded version 0.4. It appears I only half-lucked out: Dragon Age II does use GFF V4.0, but not ERF V2.x which I had programmed but never uploaded. I have now, along with a fix for circular structure definitions, like RIM structures in CIF files.

As more information on ERF V3.0 is discovered, I'll add support, but all I've got now is the ability to read the table of contents.

As always, I hope someone finds a use for either the editor or the python modules, and I'm always open to criticism and suggestions.


It's good to know that it's not a different GFF format after all, and all Bioware did was change the ERF format a bit.

#6
Mephales

Mephales
  • Members
  • 83 messages

daywalker03 wrote...
It's good to know that it's not a different GFF format after all, and all Bioware did was change the ERF format a bit.


Well, there's no guarantee that there aren't any GFF V5.0 or something lurking in the new ERF V3.0 archives... But the .tlk and .cif files that aren't in an archive are V4.0.

Modifié par Mephales, 26 février 2011 - 09:49 .


#7
Mephales

Mephales
  • Members
  • 83 messages
Turns out the profile.dap of DA2 is GFF V4.1, so there's both a new ERF format and an upgraded GFF format for us to work on reverse engineering until we get documentation.

#8
Mephales

Mephales
  • Members
  • 83 messages
Version 0.41 is out... I've added basic support for GFF V4.1 (Read-only) and support for opening files in ERF V3.0 archives, thanks to Thought Process' updates to the ERF toolset wiki article. I expect the ERF V3.0 support is quite incomplete, however. I also added the ability to export single files from ERFs, and plan on adding the ability to export all files at once.

#9
daywalker03

daywalker03
  • Members
  • 357 messages
Very good work, Mephales. I thank you, and I'm sure others will as soon as they notice what's happening.

#10
Mephales

Mephales
  • Members
  • 83 messages
I've uploaded version 0.4.2, it primarily adds better support
for V3.0 ERFs, an export all option, and a place to enter passwords for
your protected ERFs.

daywalker03 wrote...

Very good work, Mephales. I thank you, and I'm sure others will as soon as they notice what's happening.


The real thanks for these recent updates goes to Thought Process on the
Builder Wiki, who's been updating the ERF article with better specs on
V3.0. I'm just implementing what is described there.

Modifié par Mephales, 02 mars 2011 - 01:49 .


#11
daywalker03

daywalker03
  • Members
  • 357 messages
True; I guess I should thank you as well, Thought Process.

#12
Thought Process

Thought Process
  • Members
  • 191 messages
You're welcome. My work on the ERF article is pretty much done for now, not much left to update (aside from perhaps the zlib header stuff).

#13
Eshme

Eshme
  • Members
  • 756 messages
Ooooh, anything to be done about the filenames? They seem to be garbage at best :P

#14
Mephales

Mephales
  • Members
  • 83 messages
If you're referring to the FVN{####.####}.dat that my program spits out, not much. Those are files without filenames, and those are just the filenames I'm generating using the FVN hashes in the toc entry. I am thinking of generating a list of file extensions used by Bioware and their FVN32 hash, so that at least it would be .ncs or .are instead of .dat.
Maybe the released game will include more of the filenames. Otherwise, the only real option is writing a tool that could scan through all the files looking for filenames, generating their FVN64 hashes, and storing those in a database that could be used to lookup the filename later.

#15
Thought Process

Thought Process
  • Members
  • 191 messages
Like this? :)

http://svn.gib.me/pu...s/Dragon Age 2/
http://svn.gib.me/pu...gon Age 2 Demo/
(see files in subdirectories)

http://svn.gib.me/pu...efault.typelist
(pretty sure this list of types is now complete for all ERFs in the demo but I have not been stingy about checking everything)

http://svn.gib.me/pu...export.filelist

Edit: also, it's FNV, not FVN (Fowler–Noll–Vo).

Modifié par Thought Process, 02 mars 2011 - 10:36 .


#16
Mephales

Mephales
  • Members
  • 83 messages
Uh, yeah, that would be exactly the sort of thing one would need to generate hashes for a reverse lookup. Is there something you'd like to share with the class? :)
And here I was thinking that I was going to sit down and play some DAO, and you go and find more work for me.

#17
Thought Process

Thought Process
  • Members
  • 191 messages
That's my in-progress set of lists for the demo ERFs, not sure what else you mean about sharing?

#18
Mephales

Mephales
  • Members
  • 83 messages
Just a comment on how much you seem to have divined.

#19
daywalker03

daywalker03
  • Members
  • 357 messages
Oh wow; talk about hitting paydirt. Those will be helpful to a modder I know that wants to have a DA2 version of her mod ready as soon as possible.:o

#20
ladydesire

ladydesire
  • Members
  • 1 928 messages

daywalker03 wrote...

Oh wow; talk about hitting paydirt. Those will be helpful to a modder I know that wants to have a DA2 version of her mod ready as soon as possible.:o


Isn't that the truth, good buddy. I'll just bet those files, once unlocked, will be just like the ones for DAO, as far as format goes.

#21
Thought Process

Thought Process
  • Members
  • 191 messages

Mephales wrote...

Just a comment on how much you seem to have divined.

It's fairly simple, but I've been doing this stuff for a long time, so it comes pretty naturally to me. :)

#22
Mephales

Mephales
  • Members
  • 83 messages
Well, 0.4.3 is up. Only major change is that it now recognizes that list of extensions Thought Process has created, along with the filenames from his lists as well. Tomorrow I'll add options for the user to load filenames and extensions from text documents, for now it's stuck with the ones it's shipped with.

#23
daywalker03

daywalker03
  • Members
  • 357 messages
I'm assuming that the cnv extension has replaced dlg as the default for conversations, based on what Thought Process has done; it will be interesting to see how those compare to the "old" format, as far as constructing them in the toolset goes.

#24
Mephales

Mephales
  • Members
  • 83 messages
You should try opening a talktable... I'm not sure what they gone and done there.

#25
Thought Process

Thought Process
  • Members
  • 191 messages
I will take a look at the new talktable format after I finish writing my GFF serializer. :)