55 réponses à ce sujet

[tmp7704]

Trying to get a GFF file reader working for a mesh importer/exporter but running into a wall with pretty brief documentation.

as far as i can tell (for v.4.0 of the format)

* DataOffset in the file header points to beginning of Raw Data Block
* FieldOffset in individual struct templates (in the Struct Array) points to field definitions for this particular struct, in Field Array
* Index in individual fields specify offset from the beginning of data for the struct in question stored in the Raw Data Block

the problem -- how do i tell where the data for struct in question starts in the Raw Data Block?

There doesn't seem to be anything that'd specify that, and seeing things like data for the top-level struct apparently stored near the end of the file when the wiki says "The data for the top level struct is stored at the beginning of the block" is a bit of "wtf" moment...

[tmp7704]

Bump because the forum moves pretty quick :/



figured out the thing with some of data for the top-level struct being at the end of file is caused by ECString for "Name" field being stored as reference even though the field has no reference flag set. Is this the default behaviour for the strings in GFF, or just some sort of obscure exception to make things more tricky..?

[tmp7704]

Another bump and another issue

The "mesh" structures contain amongst other things a list of "chnk" structures. Struct Array for *.msh files goes as follows:

"mesh"
"decl"
"bnds"
"strm"
"chnk"

... the problem: the list of structs in the "mesh" structure references index of "32" into the Struct Array which if anything would point to "bnds" template (each template in the Array being 16 bytes large). So uhmm.. what gives and how is the index into the Struct Array supposed to really work?

[tmp7704]

Bump.

poked around some more and checked a mesh file with more than one chunk. For this particular file (ef_har_hb3_0.msh) the list of chunk structs goes: 2 (number of chunks), 32, 416

One thing i can deduce from it is, these are not offsets into the Struct Array because the whole Struct Array is something like 80 bytes large. They aren't also offsets into Raw Data block seeing how the data for first chunk begins at offset 96. And they aren't offsets into Field Array either, with each field being 24 bytes large.

So, the question is, again:

given a list of structs is stored as: "number of structs", magicnumber1, magicnumber2, magicnumber...n

... how do i handle any of the magicnumbers stored in the struct list, to retrieve from it the template of struct from the Struct Array i should use, and the starting point in the Raw Data block with the data for this particular struct? this is pretty much just extension to the problem from the first post.

Afraid without getting past this hurdle it's end of the road at this point. Srsly, any Bioware guys out there (or anyone really) with better insight into this GFF business than what the vague Wiki page offers?

Modifié par tmp7704, 19 décembre 2009 - 04:45 .

[DarthParametric]

You should talk to tazpn about it and maybe look at the source code for his importer/exporter - http://social.biowar...ject/218/#files

[ladydesire]

Are you trying to generate an msh file directly from the data coming out of a specific 3D modeling program and bypass the need to process an XML file? I know NewByPower has done that for Blender, but I'm not sure it's possible for other modeling programs.

[tmp7704]

That'd be part of it, yes. If i can parse the GFF files in general (and then more specifically, the structures related to the 3d models) then it should be possible for a 3d program to both read and write such files directly with help of specialized plugin, without the need for middle step which is export to *.xml files.



Addmittedly i could skip most of this headache doing just exporter for 3d data to the *.xml format, but if i can also get the import working then that could give insight into how the default utilize weight maps for bones and such, as well as some starting point for modifications.

[FollowTheGourd]

I admit, it doesn't make sense, but that couldn't possibly be the StructType, could it? I don't think that was really important in GFF v3.2, but just one more thing to rule out. I doubt it myself, though; it still wouldn't make sense to be in a list index.

Modifié par FollowTheGourd, 19 décembre 2009 - 11:09 .

[elys]

I coded internally a GFF 4 parser for my GDApp project which load the a GFF file into memory  in structured lists, raw datas etc.

But I've made a general only loading GFF parser, not saving because since I only need to create GDA files, I took coding shortcut for generating GDA(gff) files.

What coding language are you using for your project?

And about ECString, yes they are always stored as reference.

Modifié par elys, 20 décembre 2009 - 09:07 .

[tmp7704]

I'm using c++ for it. Well, at this point it looks i might just try to use daolib from tazpn's project as a black box to deal with GFF and just build the actual plugin on top of that.

About the ECString, are you sure? Oddly enough in the mesh files i tried to parse the Name string of the top-level struct was a reference, but one in "chnk" structure wasn't -- at the point indicated by the field it was just length of string and then the string characters. It's possible i just picked the wrong spot for where i thought the child struct would begin, i'm still pretty much in the dark about how that particular part is calculated.

Modifié par tmp7704, 20 décembre 2009 - 10:26 .

[elys]

Yes as far as I've found:

Each ECString is declared with the following FieldType record:
TypeID: 14 (0x000E)
Flags: 0

With the Field Index pointing to the string reference, which points to the string (size+characters).

If you found an ECString not declared by reference, what was the FieldType.Flags ?

What's the daolib project ? Do you have any link to it please ?

Modifié par elys, 20 décembre 2009 - 10:51 .

[tmp7704]

It's regular value of 14. But it could be easily just me going backward about the whole thing like i added in the edit, i.e. what i took as the beginning point of the child struct in fact isn't, and so in my lookup i'm hitting the string itself rather than the reference which is pointing to it.

It's this one thing mainly that's making it quite impossible to proceed (if i don't just use the daolib instead) --- can't figure out how the values for list of structs are supposed to be interpreted to get the struct type and the starting point for the structs out of them.

edit: daolib is part of tazpn's project linked here earlier: http://social.biowar...ject/218/#files 

Modifié par tmp7704, 20 décembre 2009 - 10:50 .

[elys]

Ah thanks. Too bad I did not know about daolib before, I would have not wasted time making my own xD

Why don't you use daolib then , so you could focus on the mesh code stuff directly ?

Modifié par elys, 20 décembre 2009 - 11:01 .

[tmp7704]

Well that's the plan at the moment. The struct problem is just an irritating "but how the heck is it actually working" thing that i wouldn't mind knowing Also, understanding how it's supposed to work can be of help if i run into some problems with it later down the road.

[tmp7704]

Hmm well, poking deeper it turns out the save part in daolib is a stub at least at the moment, so if i want to make something fully functional i have to figure out this struct storing/reference business anyway.



Elys, since you've written a parser yourself i'd take it you do understand how it's handled? Would it be possible to shed some light on it..?

[Tierrie]

@Elys, did you manage to figure out the data structure of the GFF v4.0 files?



@tmp7704, I am also starting to take a look at the the structure - I wrote a GFF v3.2 parser and attempted to parse the file anyway - the header is entirely different and there appears to be additional version information where the struct offset used to be.

[Werefox009]

I've been working on a python parser for GFF files (to do a level hieghtmap tool) and have nearly got a working parser, with only the generic lists left to do. I've been working from the http://social.biowar...t/index.php/GFF page, and will add some updates once I have completed writing a GFF file using my parser.



The file is broken up as:



Header

Structure list (the first is the structure representing the file itself, so there is always atleast one per file)

Field list ( structure 0's stored first, then structure1's and so on,

Raw Data Block



The structure at position 0 in the structure list contains all the other structures and fields etc, the actual start position of the data is at the Raw Data Block offset. Each field in the structure can be accessed using the offset in the appropriate field list the current structure.



There are several different types of field data that can be found within a structure, inline-value, list of specific type, list of structures, generic lists (essentially a list that can contain any type of information), structures and finally references. As it stands in my test data I haven't hit a reference field, and I am currently stuck on parsing generic lists.



A structure field for a list contains a 4 byte offset/reference from the start of the current structure position. The data there is then a 4 byte unsigned integer representing the size of the list, followed by size * whatever type the list consists of. If the list is empty, the 4 byte offset/reference will be 0xFFFFFFFF.



A list of structures is much the same, with the exception that instead of the field type flag representing a specific type, it represents the structure array index for the structure that makes up the list.



ECStrings are considered a list of 2 byte characters, and follow the same setup as a list. The difference is that the field flag does not indicate that the ECString is a list, so you will need to watch for fields with a type of 14 (that aren't a structure type)



Generic Lists (lists where the field type is 0x0000FFFF) appear to follow the same as a list, except that it is a list of references. However this part is a work in progress/hack at it until it works scenario, and I may be well off base at this time.



There are reference fields, but I have not had a specific example of this yet, but they seem pretty straightforward on the GFF file format web page, and in-line value fields which you can get to by moving the field array index value from the start of the structure in the datablock.

[Adinos]

I had to write a complete GFF parser for my model viewer, so If you get stuck on something I probably have an answer for you - however, my source code is currently 3000 kilometers away from me, so until I get back home I will not be of much help

[elys]

@tmp7704: Yes I will be happy to help if I can. Just that these end-year days are rather busy with parties and alike for me, so I won't really use the forums until next year 
Hopefully Adinos, Tierrie and Werefox009 are also around with knowledge.

@Tierrie: The Toolset Wiki: GFF has been quite helpful to me, even if I still had to figure some stuff by analysing GFF files using an hex-editor to understand how it stored all the stuff.

Modifié par elys, 23 décembre 2009 - 04:13 .

[Tierrie]

@Adinos, if you wrote your parser in C, I would really love to see it. I am working on parsing PLO (plot) files. There are lots of references and list of structures in this one.



@Werefox, so in the case of list of structures, FieldType is the index of the array. In that particular case, what does List size represent?

[Tierrie]

@Werefox, as far as I can tell, my ECStrings are always flagged as reference and list.

[Werefox009]

Hmm I probably didn't explain that well at all. Say we have a file that has several entries in the structure array portion of the file, and the first field in the top level structure is actually a list of a specific type of structure. The file details as below:



Header Details

Structure Array

- [0] top level

- [1] MESH structure

- [2] PROP structure

Field Array

- Structure[0].Field[0]: list of structures. flags: 0xC000, type: 0x0002, label=3000, index=0

- Structure[1].Field[0]: ...

- ...

- Structure[2].Field[n]: ...



Raw Data Array.



Starting from the begining of the raw data array we know that it is structure[0].field[0]. Checking the flag id for structure[0].field[0] we know that it is a list, and that it is of type structure. The type for structure[0].field[0] indicates that it is 2.



Now if this was a standard list, then the value of 2 would represent a list of unsigned 16 bit values. However because the structure flag is also present, this instead means that it is a list of structure type identified in the structures array at index 2, or in this case a list of PROP structures.



We advance the file position structure[0].field[0] index bytes from the start of the structure to get to the start of the reference for the list proper. Read the next 4 bytes for an unsigned offset from the start of the current structure, and check to make sure it is not 0xFFFFFFFF (or the empty list reference). Advance to the indicated position in the file.



For all lists the next 4 bytes are an unsigned count of the number of items in the list. All the items in the list follow from the very next position. I'll stop there as I think I just found an issue with my list reading code for specific types structures, and if i have then it will just confuse the issue more



Cheers,

Werefox.

[Werefox009]

Good to know about the ECString, i'll have to double check what I am doing for it. My parser is still very much a work in progress, and not 100% functioning yet

[tmp7704]

Werefox009 wrote...

Now if this was a standard list, then the value of 2 would represent a list of unsigned 16 bit values. However because the structure flag is also present, this instead means that it is a list of structure type identified in the structures array at index 2, or in this case a list of PROP structures.

Oh gods. And suddenly all the other pieces fall into place and it starts making sense again. Plugged it into the code and it appears to handle the struct lists correctly now or at least doesn't crash and burn horribly in the process.

Thank you, this was most helpful.

can't believe i completely overlooked the field type would still hold that data, i got too wrapped up in the content of the list instead *kicks self*

Modifié par tmp7704, 23 décembre 2009 - 10:15 .

[Tierrie]

I take back what I said about list & reference flags being set in the case of CString. I made an error in the bitmask. They are NOT set.