11 réponses à ce sujet

[DarthGizka]

I've hit a few brick walls with the usual means of getting detailed information about the game - poking around in the toolset, querying the toolset database via SQL, running around in the game and measuring stuff with or without the help of special scripts.

The first two methods give only incomplete answers about an outdated snapshot of the game excluding all DLC and mods, the third is extremely time-consuming and still rather limited. None is really viable for things like automatic validation of ability/spell/item tables against the actual game data, or finding out which mod b0rkens what.

That's why I'm looking at the possibility of analysing the actual game data files, with or without the expedient of dumping parts of it into an SQL database like the one that comes with the toolset. I'm also looking at ways of peeking into NCS files as well.

It seems likely that at least some of that work will carry over to DA:I, allowing us to hit the ground running when the new game comes out. Does anyone know specifics already?

If DA:I uses BioWare's standard resource file formats and some evolution of NWScript then it would be worth it to invest some time into making solid tools rather than the throwaway duct tape & chicken wire affairs one tends to use in spelunking. It would also make it worthwhile to get DA2, to have an extra set of raw data that is closer to DA:I than DA:O is.

I haven't been able to locate a copy of nwnnsscomp that works with DA:O's files; the ones I found were all for NWN or KotOR and choked on a script.ldf renamed to nwscript.nss despite having been released as recent as 2012. The source code then turned out to be exactly the same as a version timestamped 2004-06-16.

The most recent bytecode documentation still seems to be Torlack's from way back when, although nynaeve has been doing some interesting work as recently as 2010 (an MSIL backend for NCS pcode to add jitting to the NWScript VM) and the site may harbour a lot more interesting info.

The newest version of DeNCS I found is from 2006.

The Open Knights effort seems to have fizzled out in 2006 or so.

Xoreos seems to be active still and their specs directory is a veritable treasure trove.

Be that as it may, it should be possible to get nwnnsscomp into working shape or at least to produce a functioning two-way assembler/disassembler. I've got some experience with disassemblers (mostly x86), and I wrote a fully functioning decompiler for FoxPro/VFP in VFP itself when I was new to it and had to get proficient in a hurry. The thing started as an attempt to clarify some fuzzy information in the documentation and sort of grew from there (recompilable decompilation being nice for the automatic verification of stuff). So that's something I could tackle if it is confirmed that DA:I uses NSS/NCS as well.

I guess with Mephales' pyGFF (BSN project, topic) we have GFFs pretty much sewn up, so we'll be covered on that front... The linked version is 0.5.5b, dated 2011-04-27.

One thing that is a bit of an open question is the referential structure - i.e. where the game goes looking to find what resource and when, which override directories have precedence when, which bits of code are called by the engine when, and so on. That information seems to be scattered high and wide over the toolset wiki.

 

It's something that mod manager programs should be aware of in order to be able to warn about possible conflicts and for merging/optimising mod installations, but I couldn't find much in that direction.

[DarthParametric]

I wouldn't hold your breath on a lot of file format similarity between Inquisition and DA1/2. Much as was the case with Mass Effect and the use of Unreal Engine, the move to Frostbite likely necessitated a complete change in the way the game was put together. I'd suggest waiting until the game is released before getting too carried away. You'd probably be better served looking at Battlefield modding, as chances are there will be some degree of crossover.

[DarthGizka]

I see what you mean. The virtual file system is likely embedded deeply in the engine, and thus the file formats. Not to mention the production pipeline...

 

Scripting might be a different story, especially as I couldn't uncover evidence of a 'native' scripting system like e.g. UnrealScript.

 

Official comments regarding mods and DA:I seemed to contain more rubbish than anything else. Never mind modding support - it would probably be sufficient if they didn't actively try to lock things down, by making it essentially illegal to run a modded game.

[DarthParametric]

There won't be a toolset, that's for certain (and I wish devs would just be upfront and honest about it, instead of trotting out the tired "we're investigating it" - they're still "investigating" the DA2 toolset as well I gather). It's a console game first and foremost, which means file format streamlining to suit those platforms, some of which was seen in DA2. I would hazard a guess that EA will probably take a dim view of modding, lest there be any blowback onto Battlefield. I would be unsurprised if we see DAI modding threads here wind up deleted if they manage to peek a little too far under its skirt, although it will be interesting to see what happens on other sites like Nexus and Xentax.

[DarthGizka]

The 'disappearing thread' thing can be spooky...

[DrMcCoy]

 If you'll excuse the thread necromancy...

 

If DA:I uses BioWare's standard resource file formats and some evolution of NWScript

 

Now that DA:I is out, what's the word on that? (I very much doubt it does, though.)

 

Xoreos seems to be active still and their specs directory is a veritable treasure trove.

 

Just as a heads-up, I split the docs into their own repo. So the specs directory is now here: https://github.com/x...ee/master/specs

 

And I'd really like to expand that repo there. Maybe even make it the (semi-)canonical place for BioWare-engine specs, unless you're now all yelling "bad idea".

 

Right now, file format and general engine-level information for the different games is spread across many different websites. And when one should die (like Torlack's old site or nwn.bioware.com), a part of the info is lost. The xoreos wiki too could just die and vanish. A git repository however is self-contained and easily copy- and transferable.

 

If there's anybody interested in that, I'd welcome discussion on how to organize this, links to where specs currently can be found, offers to help, pull-requests, ...

The GitHub issue tracker would probably be the best place for this, I think?

 

Likewise, if there's further modding tools out there, with sources and a proper license, I'd like to aggregate those too. I'm kinda saddened to see how quickly those get lost, and how often those are binary-only. But I am one of those pesky free software zealots.

 

About NWScript. Is there info out there what changed after NWN? I have the NWN bytecode down in xoreos, I think, and IIRC KotOR should work too. But were there any new opcodes or anything in NWN2 and KotOR2? They did add a few features, after all, like being able to pass parameters to dialogue scripts. And I haven't even looked at what Jade Empire, or the Dragon Ages do for scripts. Really, if there's anybody who knows about this, or is willing to research, and could do some write-up for methe community, that would be most appreciated.

[DarthGizka]

DA:I seems to be using Lua. This is good news in the sense that Lua is a fairly mature* general script language with existing decompilers etc., which may allow plugging major functionality into the game fairly quickly. And it also allows development and testing of major script systems in isolation, which is a major improvement over the development cycle with DA:O. Once you get past the bizarre syntax it offers quite a bit of power and convenience, and this could allow much more interesting and complex systems to be scripted with less effort than in NWScript.

 

A central hub/nexus for intelligence relating to Bioware game engines is a brilliant idea. If GitHub allows chaotic (unplanned/unsystematic) cooperation and contribution like the NWN and Dragon Age wikis then I'm all for it; otherwise it might rest rather heavily on the shoulders of a few giants.

 

Yes, binary-only contributions are a bit of a waste since the hard work their authors put into them is essentially lost once the binary cannot be used for its direct purpose anymore...

 

*) i.e. now, after 'only' two decades; see how long it took for bitwise operations to arrive

[DrMcCoy]

DA:I seems to be using Lua.

 

Does the game data contain Lua sources or Lua bytecode? Because Lua bytecode is pretty icky to deal with: it's basically bound to one specific CPU architecture and the specific Lua version. That's a problem xoreos already has with The Witcher, or rather will have in the future.

 

If GitHub allows chaotic (unplanned/unsystematic) cooperation and contribution like the NWN and Dragon Age wikis then I'm all for it; otherwise it might rest rather heavily on the shoulders of a few giants.

 

Well, yes and no. Everybody can fork the repository, and work separately and chaotically. For it to be merged into the main / "official" repository, someone needs accept the pull requests and possibly manually resolve merge conflicts that couldn't be resolved automatically. (And you can also do that outside of GitHub by throwing a patch into a mail; git has commands for that. This is what the official Linux kernel development does.)

 

So it's not quite like a wiki, since you still need someone to "approve" things before they go into the main tree. And you can't just quickly pop onto a website and make a change. You do need to at least clones the git repository. And yes, the barrier of entry is higher.

 

Still, I for one think that's the best of both worlds: you can have a lot of people working on it, but you can still have a structured and unified end result. And, again, I really don't like that your common MediaWiki is a black box you can't easily export, copy or transfer.

 

I think those advantages outweight the disadvantages of a higher entry barrier and need for people to coordinate a bit, but YMMV.

 

Yes, binary-only contributions are a bit of a waste since the hard work their authors put into them is essentially lost once the binary cannot be used for its direct purpose anymore...

 

Especially if the binary is Windows-only and you happen to not run Windows. Or if, like me, you're not actually interested in the tool itself, but in the information contained within. I did already sick Reflector onto a tool written C#, and IDA on another tool... Feels quite silly.

[DarthGizka]

It's even worse if the binary is 64-bit and you have the crippled version of IDA like I do (my support plan ran out a few months ago and so I can't upgrade anymore)...
 
If you are willing to give it a go - why don't you post a link, and perhaps a link to a good tutorial so that everyone can get up to snuff.
 
As regards DA:I and Lua: I've heard rumours about source code being at least partially present, and in the case of initfs_Win32 there are even samples on the 'net. However, I haven't started spelunking myself since I'm still busy with my first playthrough. So far I've only poked around a bit and tried to make sense of file formats.
 
My trusty Fox can already parse some things, like ProfileOptions, .toc and .sb. Here the next step will be making a little scanner that checks all hypotheses against the full game data (C++, since FoxPro is a tad slow for such bulks of data).
 
Here's some sample output for .toc and .sb:

# Update/DeluxeContent/Data/Win32/DeluxeContent.toc
0000022C 82 >> 00000101 (-> 00000330) 81 02
0000022F    01 bundles >> 000000B2 (-> 000002EC) B2 01
0000023A       82 >> 00000058 (-> 00000294) 58
0000023C          07 id (38) Win32/da3/party/mount/blueprintbundles/unicornbogbundle
00000279          09 offset 00000000:00000010
00000289          08 size 00004CBF
00000293       00 << 00000058 0x82 @ 0000023A
00000294       82 >> 00000055 (-> 000002EB) 55
00000296          07 id (35) Win32/da3/party/mount/blueprintbundles/redhartbundle
000002D0          09 offset 00000000:00004CCF
000002E0          08 size 00004E4F
000002EA       00 << 00000055 0x82 @ 00000294
000002EB    00 << 000000B2 bundles @ 0000022F
000002EC    01 chunks >> 00000001 (-> 000002F6) 01
000002F5    00 << 00000001 chunks @ 000002EC
000002F6    06 cas 01
000002FC    07 name (14) Win32/deluxecontent
00000317    06 alwaysEmitSuperbundle 00
0000032F 00 << 00000101 0x82 @ 0000022C
00000330 *EOF*

# Update/DeluxeContent/Data/Win32/DeluxeContent.sb
00000000 82 >> 00009B1C (-> 00009B20) 9C B6 02
00000004    01 bundles >> 00009B0F (-> 00009B1F) 8F B6 02
00000010       82 >> 00004CBB (-> 00004CCF) BB 99 01
00000014          07 path (38) win32/da3/party/mount/blueprintbundles/unicornbogbundle
00000053          08 magicSalt 7065636D
00000062          01 ebx >> 00001C1A (-> 00001C83) 9A 38
00000069             82 >> 00000096 (-> 00000102) 96 01
0000006C                07 name (50) da3/sound/assets/foley/abilities/dropdown/fly_csm_humanoid_metal_dropdownmed_01
000000C3                10 sha1 60C783F39C0551EEAF8D800DEA2632783AF507B5
000000DD                09 size 00000000:000004EC
000000EB                09 originalSize 00000000:00000930
00000101             00 << 00000096 0x82 @ 00000069
00000102             82 >> 0000007B (-> 0000017F) 7B
00000104                07 name (35) da3/sound/assets/foley/mounts/fly_mounts_horse_brake
00000140                10 sha1 B4DAF91D00BE448E0DA0BB18B42C1158681305AB
0000015A                09 size 00000000:000004D5
00000168                09 originalSize 00000000:00000920
0000017E             00 << 0000007B 0x82 @ 00000102
...
00009A9E             82 >> 00000020 (-> 00009AC0) 20
00009AA0                08 h32 C6E038C4
00009AA9                02 meta >> 0000000F (-> 00009ABF) 0F
00009AB0                   08 firstMip 00000003
00009ABE                00 << 0000000F meta @ 00009AA9
00009ABF             00 << 00000020 0x82 @ 00009A9E
00009AC0          00 << 000001B1 chunkMeta @ 00009903
00009AC1          06 alignMembers 00
00009AD0          06 ridSupport 01
00009ADD          06 storeCompressedSizes 00
00009AF4          09 totalSize 00000000:002FE963
00009B07          09 dbxTotalSize 00000000:002FBDE8
00009B1D       00 << 00004E4B 0x82 @ 00004CCF
00009B1E    00 << 00009B0F bundles @ 00000004
00009B1F 00 << 00009B1C 0x82 @ 00000000
00009B20 *EOF*

For ProfileOptions I don't have a text dump since my script reads it directly into a cursor (temporary table in Fox), for direct querying via SQL. For savegames I have yet to dig into the "FB SAVE" blocks. The decription block looks like this:
 

00000016 FBHEADER 00 01 00 00 00 10

0000000E 72677992 0006 Alayna               // character name
0000001A A06F6E92 0001 0                    // character number (count starts at 0)
00000021 8A71AE06 0001 1                    // ???
00000028 3DAD9C97 000D 385628.593750        // == 107:07:08 h (time played in seconds)
0000003B D8BD15B6 0024 19E2E45B-5636-272C-F5A6-EAA911C246EB  // 128-bit character id
00000065 FB9770E1 0001 1
0000006C 032FCAE1 0002 17                   // level
00000074 F0BD21A5 0013 POS-1.7;91.6;-41.1;  // position
0000008D 340F505F 0006 187535               // ???
00000099 B82F852F 0011 Cradle of Sulevin
000000B0 9F1F99B3 002A UI/Static/SaveScreens/level_RevenentChurch
000000E0 892D839A 0053 DA3/Layouts/ThematicLevels/the_RevenentChurch/the_RevenentChurch/the_RevenentChurch
00000139 B08AAA39 000C 212284536181         // 2014-12-04 22:43:01 (seconds since 0001-01-01 00:00)
0000014B B0F50985 0006 856038               // constant (perhaps player-specific)
00000157 F1EA4603 0001 1                    // ???
0000015E 47FB864D 00CE                      // loaded add-ons, with ~ as separator/terminator
   SP_SLKEY_802@Dragon Age™: Inquisition Deluxe Edition Items~
   SP_SLKEY_805@Flames of the Inquisition Armor~
   SP_SLKEY_804@Flames of the Inquisition Armored Mount~
   SP_SLKEY_803@Flames of the Inquisition Weapons~

00000232 7DD2A03B                           // checksum?

Just to get things started and programmer's fingers itching...

[DrMcCoy]

It's even worse if the binary is 64-bit and you have the crippled version of IDA like I do (my support plan ran out a few months ago and so I can't upgrade anymore)...

Yeah, I too only have the Standard version sans 64-bit support. To be fair, my license is paid for by the ScummVM project budget, and I don't need 64-bit support for my work on ScummVM.
 

If you are willing to give it a go - why don't you post a link, and perhaps a link to a good tutorial so that everyone can get up to snuff.

Errr, what exactly now?

Using the xoreos-docs git repository as a specs aggregator?
Well, the repository is here.
For tutorials, I guess there's the GitHub help pages and the Git book if you want to dive in deep.

I'm willing to step up as a coordinator, unless someone else wants to help. I'm not really sure how to best organize things yet, though, only that I'd probably want a "sane" directory structure and single topics as Markdown files. Any thoughts?

I'd probably want to convert the already existing HTML and PDF files to Markdown, too.

Or did I just misunderstand where you were going?

For the specifics of DA:I, I have to bow out. I don't actually have DA:I, so I'm not sure what I'm looking at there. My interest there is more..."academic" in nature, because I'm curious about how BioWare-y the new file formats are. Sorry.

[DarthGizka]

Well, I thought you envisioned some kind of hub, a central goto/linkage point for Bioware Arcana. The repository is pure gold but ongoing work will often be focussed on low-grade ore, which has to be refined and worked a lot before it can be added to the stash. So it would be linked rather than incorporated, during the early stages. There's a lot of value in this linkage, to bring more people in contact with information and active work groups.

Also, gems like nynaeve's work on a JIT engine for NWScript need to be collected and linked...

The data dumps were not aimed at the repository - I posted them in order to get more seed material for the DA:I effort into circulation.

[DrMcCoy]

So it would be linked rather than incorporated, during the early stages. There's a lot of value in this linkage, to bring more people in contact with information and active work groups.

I have no idea what you're talking about. o_O

You can both set links in Markdown (see for example the README.md (in raw), and you can link from the outside to it, as I have shown just now.

It's Markdown, plaintext that can be read as is, or be parsed to HTML. If we're properly structuring it in directories and Markdown files, it's no different than a wiki in organization.

Or what do you mean?