15 ответов в этой теме

[SkywingvL]

I'm making available some of the supporting libraries that the Client Extension uses in order to interface with many of the game's data files.

This library may be of use to programmers and developers of NWN2 tools and utilities. You can get a copy of it here:

Public NWN2Dev library download link

An example tool (a simple MDB renderer against GDI) that functions as a demonstration of how to use the libraries is also included.

The library is set to be built with the Windows Driver Kit (WDK). It will also build with Visual Studio (VC++), but I don't maintain a VS development environment for the project any more, so you'll have to set up the projects yourself. The WDK's freely downloadable and self-contained, and a link to it is included in the readme file.

Including the blurb from the original post describing the library:

The library consists of heavily commented C++ code that provides a suite of file format parsers for various NWN2 formats:

- TRX format (all packets/records are fully decoded), including the complete area surface walkmesh and navigational node format. Meshes and navigational data are deserialized to a convenient set of data structures suitable for drawing or interacting with the walkmesh for pathing traversal. A partial set of APIs for interacting with the walkmesh structures (such as visibility and straight path computation) are also included.
- MDB format (all packets/records are fully decoded). Meshes are deserialized into convenient data structures suitable for drawing. APIs for performing C2/C3 collision mesh intersection in the same fashion as the game engine does are provided.
- TLK format (localized string lookup).
- 2DA format (all 2DA format features except the 2DA default value option -- which seems to not be used by NWN2 -- are supported).
- GR2 format, in the form of a wrapper for tazpn's granny2 reading code for skeletons, plus various fixes to tazpn's underlying granny2 reading code. Eventually, animation reading will be added.
- DDS images, for simple 32-bpp DDS images. The DDS reader was designed only to support enough DDS features to process the inline texture blending maps embedded in TRX files, although other 32-bpp simple DDS images could be read via it.

At present, only readonly access to these resources is available; however, as the resource formats are generally (with the exception of GR2) completely deserialized, all of the information necessary to create writer APIs is available. I have not done so yet as my projects have not yet required writing to these files, only reading.

The NWN2 datafile lib also includes a complete (readonly) implementation of an NWN2-compatible resource loader. The resource loader supports the following features:

- Directory mode file support
- ERF file support (MOD, PWC, HAK, ERF, ...)
- Zip file support (indexes all ~100K of the in-game zip resources in on the order of 200-400ms on bootup, as opposed to the 3-6 seconds that the NWN2 native resource loader burns searching through zip files at game initialization)

The resource manager provides both direct in-memory access APIs to read resource files, as well as optimized APIs for pulling resources files out to disk and allowing them to be accessed via any standard file I/O API.

A search strategy compatible with the game's native resource manager has been implemented based on the original BioWare Aurora documentation and further observations of the NWN2 engine behavior. The search strategy is believed to be complete (or nearly so) and can successfully load custom content with the correct priorities in multiplayer PWs, to the point of being able to load custom models, 2DAs, etc.

A uniform interface for accessing resource files is provided (uniform regardless of whether they are backed by directory mode filesystem files or embedded in ERFs or compressed in Zip files).

The Client Extension internally uses this library for all its resource loading needs, including model/terrain/game rules processing support.

If there are any projects which could make real use of such a library for purposes of custom content tools creation or the like, I'm open to doing the work necessary to release a version of the library for public consumption.

The code is currently written for 32-bit and 64-bit Windows, although it should be trivially portable to any other operating system. Note that granny2 reading requires calling a granny2 support function to decompress the .gr2 files to their raw format and thus granny2-related features are regrettably tied to 32-bit Windows.

Core dependencies include a zlib, the minzip library shipping with zlib in the contrib distribution (plus some minor changes for significant performance improvements in zip enumeration), an C-interface internal math library, granny2.dll (only if .gr2 support is desired), C++ STL, and basic file I/O functions and directory enumeration APIs (memory mapped I/O used in some cases, though the code is trivially portable to operate without memory mapped I/O and could easily function with standard C CRT file I/O APIs). No MFC or other framework libraries are used.

All data file formats are validated when parsed, and as such malformed data files should not introduce crashes or other security bugs. An exception to this rule is that loops in the navigational node structures in the area surface walkmesh must be handled during pathing traversal instead of at load time, for performance reasons.

The datafile library is heavily tested under load with pageheap and is believed to provide a robust foundation.

Possibly uses for the library might include custom content creation, viewing, or manipulation tools, or certain toolset plugins.

[SkywingvL]

I have released a new version of the datafile accessor library. You can download it from the same url as before:

Click Here

The change log for this release (also included in the .zip) is:

Version 20100515-01
-------------------

- Generic File Format (GFF) files are now supported by the datafile accessor system. The GffFileReader object allows validating parsing of these data structures.

- The 2DA file reader is now compatible with broken 2DA files missing the first blank line after the header (such as creaturespeed.2da).

- More verbose 2DA load failure debugging is now available.

- MeshManager abstraction added. The MeshManager and MeshLinkage support logic allows easy typed access to all meshes instantiated by a particular instance of a ResourceManager. They are intended to allow data (such as hardware-defined resources) to be attached to each mesh, and to be easily retrieved or updated.

- All mesh objects (including the AreaSurfaceMesh) will now register themselves with the MeshManager through an embedded MeshLinkage object.

- The TrxFileReader can now be configured to skip loading of display meshes for .trx and .mdb files.

- New resource type mappings added for ERF files.

- Support for partial module resource loads has been added to the resource manager system.

- DemandResource-based template classes now behave correctly when copied.

- Fixed incorrect case sensitivity for file extensions when the resource system is loading data from raw directories via the DirectoryFileReader.

Сообщение изменено: SkywingvL, 13 Июль 2010 - 06:12 .

[SkywingvL]

New release posted (same download link).



Changes

-------



Version 20100522-01



- Builds for amd64 (64-bit) are now fully supported.



- Various warnings fixed for amd64 (64-bit) build. Note that the 64-bit build requires a 64-bit version of granny2.dll (which NWN2 does not ship with) in order to load *.gr2 files.



- SRT matrix decomposition routine added to math library.



- AreaSurfaceMesh::StraightPathExists now optionally returns the face that the straight path search terminated at (i.e. as far as we got in a straight line before we ran into something).

[SkywingvL]

A new version is available. This version includes a major new library, NWNScriptLib, which allows applications to host an execution environment for compiled NWScript scripts (*.ncs).



Unline the NWN2 implementation of NWScript, NWNScriptLib does not require recompilation of script source code on the server for parameterized scripts. This restriction was eliminated through the introduction of dynamic type management for script entry point parameters.





The same download link applies for the project.





Version 20100528-01

-------------------



- The GffFileReader was not properly handling CExoLocStrings that contained only a STRREf and no other data. This has been fixed.



- The NWNScriptLib, a complete virtual machine for the core language features implemented in the compiled NWScript instruction set (*.ncs), has been added to the project distribution.

[SkywingvL]

I'm pleased to announce another revision to the NWN2 datafile library.



This release has several major components:



- A fast, robust GFF writer (with support for easy copy-and-modify editing of existing GFF files without having to understand their full contents) is available.

- Many stabilization fixes have been made to the NWScript VM, which is now generally stable for usage.



The same download link applies.



Here's the full change log:



Version 20100612-01

------------------



- A writer for generic file format (GFF) files has been added. This writer is accessible by creating a GffFileWriter object. GFF contents can be created on the fly, or initialized from a GffFileReader.



- The ErfFileReader was returning incorrect data if the same file logical handle was read from multiple times in a row. This has been corrected.



- NWScriptVM::ExecuteScript now takes an additional default parameter, the DefaultReturnCode, which is the return value of NWScriptVM::ExecuteScript should script execution fail (or should the script not return a value). Previously, the return code in these cases was always zero.



- NWScriptVM::ExecuteScript now takes a flags parameter. This allows extended control over the script VM's behavior.



- The implementation of the CPDOWNSP NWScript instruction was not properly masking off SET_ENGINE_STRUCTURE when checking for dynamic typed parameters. This has been fixed.



- There was a typo in CPDOWNSP's implementation that interfered with string copies. This has been fixed.



- There was a bug in CPDOWNSP that prevented engine structures from being correctly copied. This has been fixed.



- The script VM debugger now prints the contents of a string on the stack in addition to the string handle.



- Script parameters were not being correctly passed to parameterized scripts that did not return a value and did not use global variables. This has been fixed.



- Parameterized scripts that modified their parameters with CPDOWNSP were not being permitted to do so. This has been fixed.



- LOGANDII/LOGORII were not always correctly removing the right number of values from the stack. This has been fixed.



- Nested script calls within the same script VM stack were not returning properly. This has been fixed.



- The executed instruction count was not being reset when unwinding through the outermost call to NWScriptVM::ExecuteScriptInternal. This has been fixed.

[SkywingvL]

Saarun noticed that the toolset's GFF reader wasn't accepting GFFs produced by the GffFileWriter.  This appears to be due to logic in OEIShared assuming a fixed (sequential) layout of the different sections in a GFF file.  (The game core itself does not have this limitation.)

The next public drop will include an option (GffFileWriter::Commit(<Stream>, <Type>, GffFileWriter::GFF_COMMIT_FLAG_SEQUENTIAL) to create a GFF that is compatible with the toolset's GFF reader code.

[SkywingvL]

Version 20100815-01 is now available.  You can get it at the usual download link.

This release includes the JIT compilation system for NWScript, and various enhancements to the GffFileWriter APIs.  If you need to create a GFF compatible with the NWN2 Toolset's buggy GFF reader, you should pass GffFileWriter::GFF_COMMIT_FLAG_SEQUENTIAL to GffFileWriter::Commit.

Full changelog follows.

Changes
-------
Version 20100815-01
-------------------
- Significant improvements to the GffFileWriter class have been made,
  including List editing APIs and a fix for compatibility with the NWN2
  Toolset's GFF reader.
- Added a just-in-time (JIT) compilation system for NWScript.  An initial JIT
  backend that emits MSIL (requires the .NET Framework) is included.  The JIT
  backend must be built with Visual Studio and not the WDK.
- Added a static analysis package for NWScript to the NWNScriptLib
  (NWScriptAnalyzer).  Credit goes to Justin Olbrantz for significant work on
  the code analysis / IR raising phase of the NWScriptAnalyzer.
- Added NWScript action data tables for NWN2 1.23 and NWN1 1.69.
- Users of NWScriptVM can now provide data table pointers for action handlers
  implemented by the script host.  If provided, the data tables can be used to
  enable static analysis to determine the real argument counts for
  parameterized scripts.
- Fixed a bug where empty strings were not properly converted to OBJECT_INVALID
  correctly when passed as dynamic parameters and interpreted as type object.
- Added support for indirecting through a standalone program (Gr2Conv.exe) to
  Granny2Lib.  This enables non-x86 builds to decompress *.gr2 files.
- Fixed alignment faults on ia64 builds in Granny2Lib.

[SkywingvL]

I've created a new sample program for the datafile accessor library. The sample shows how to use the library in order to examine (and print some information about) each Door object in a module. It's intended to demonstrate how one might put together a simple program that performs an operation on all objects of a certain type in a given module.

You can get it here: http://www.nynaeve.n...ModuleAreas.zip

This sample will be included in a future release of the public_nwn2dev package.

The BioWare Aurora documentation ( http://nwn.bioware.com/developers/ ), and GFFEditor.exe, are good places to start in order to figure out what fields mean what for a given object type's GFF intance.

Сообщение изменено: SkywingvL, 18 Сентябрь 2010 - 09:17 .

[SkywingvL]

A new version of the library has been released. This release includes three additional samples:

- The ListModuleAreas sample, which walks the area list in a module.
- The ListModuleModels sample, which queries the resource system to enumerate all model resources available in a model.
- The UpdateModTemplates sample, which demonstrates how to use the GffFileReader and GffFileWriter, combined with the resource system, to load and update data in object instances records placed into the module.

The usual download link works (as in the first post).  Here's the full change log:

Version 20101228-01
-------------------

- Fixed incorrect tint set reading and writing helper functions in the
GffFileReader and GffFileWriter classes.

- Fixed I_BOOLAND emitting OpCodes::Xor instead of OpCodes::And in the
MSIL/.NET NWScript JIT backend.

- By default, NWNScriptLib now includes action service handler names in its
data tables. The NWScript JIT backend now uses these names (if present) to
name direct fast action service handler call stubs, for easier debugging.

- Fixed RESREF parsing in the GffFileReader (security fix).

- Added NWN1 BIF/KEY processing support and NWN1 ERF processing support. To
load NWN1-style datafiles, the user must supply the appropriate new values in
the ResourceManager::ModuleLoadParams struct, and LoadModuleResources must be
used instead of LoadModuleResourcesLite.

- Fixed pathfinding intersection issues in straight path detection.

- Added mesh manager notification callouts, fix mesh manager list iterator
support (thanks to Chris Darken).

- 64-bit builds will now prefer to use memory mapped file I/O when reading HAK,
BIF, and ZIP archives. This significantly improves loading performance,
especially for ZIP archives (up to several times faster in some cases).

- Added various additional math support routines, such as cone intersection.

- Added new "ListModuleAreas" sample program that prints a list of all areas in
a module.

- Added new "ListModuleModels" sample program that prints a list of all model
pieces present in a module and its HAKs (and the default game install files).

- Added new "UpdateModTemplates" sample program that performs bulk updates of
placed object instances in a module with new data from their source
templates.

Сообщение изменено: SkywingvL, 29 Декабрь 2010 - 03:22 .

[The Fred]

OK, late reply, but "wow". Just wow. If I actually understood what all of this did, I feel I would be able to appreciate it more fully.

[kevL]

ya, i say we ask Skywing what he wants: Hall of Genius, 70 dancing virgins .. what?

[c i p h e r]

Can this be used to create a more efficient staging tool?

Right now, the toolset processes every file when preparing shared content. It also requires loading the toolset with all of its resources, which is not very efficient when simply making updates to files or adding new files to a campaign/module. It would save an enormous amount of time if we had a stand alone tool to which you could specify the file(s) to check and re-stage and possibly even upload.

In essence, an updated Worldgate client that did the work of the ADL but outside of the toolset. Just an idea.

[SkywingvL]

Yes, you could use the library to enumerate all of the haks/TRXs and other freestanding datafiles that a module depends on. You would then want to hash these and compress them with the LZMA toolkit library to stage them, and write the updated download resources XML.

[SkywingvL]

A new version of the library has been released. The same download link, at the top of the thread, will work.

Here are the changes:

Version 20110507-01
-------------------

- Fixed TimerRegistration::Stop returning the wrong amount of time left on the
timer.

- Fixed incorrect timer logic in the TimerManager relating to due time caching.

- swutil::SharedPtr<T> and swutil::SharedBuffer<T> now evaluate the deleter
for their input pointer in the constructor, should the shared state not be
created successfully. Previously, the deleter was not evaluated in this case
and the caller was responsible for deleting the input pointer. The new
behavior is more consistent with boost and C++0x shared_ptr.

- Imported Edward T. Smith's NWScript compiler with various improvements.

- Added DLL version of NWScript compiler, and NWN2 toolset plugin wrapper to
allow the toolset to use the script compiler.

- Added new compiler shell (NWNScriptCompiler) to replace nwnnsscomp.

- Added on demand compilation from source text to NWNScriptConsole.

- Added option to load base resources only to ResourceManager.

- Added UnloadAllResources API to ResourceManager.

- Fixed incorrect behavior of ResourceManager::OpenFileByIndex.

- Added ResourceManager::ForceCloseOpenHandles API.

- Added APIs to query prototype data for scripts to script compiler library.

- Added option to use Granny2 out of process even on x86 (for stability).

- Added custom "first chance" and "last chance" IResourceAccessor support to
the ResourceManager. This allows the user to easily plug in an arbitrary new
resource backend to the resource system.

- Added build system support for compiling NWScript scripts.

[SkywingvL]

A new version of the library has been released. The same download link, at the top of the thread, will work.

Here are the changes:

Version 20111230-01
-------------------

- Major improvements to the NWScript compiler, including bug fixes and new
  compiler optimizations.

- New APIs on the NWScript compiler DLL version for querying parameter
  information about symbols.

- New APIs on the ResourceManager object for better controlling module loads.

- Significant refactoring and improvements on the NWScript JIT backend, with
  support for creating and hosting "CLR scripts" that are written in .NET
  languages and can call all action service APIs exposed by the game server.

- Many other bug fixes and improvements, particularly relating to NWScript
  compilation and execution.

Сообщение изменено: SkywingvL, 30 Декабрь 2011 - 06:44 .

[FantasyDR]

I have to say this lib is awesome!!!!