116 replies to this topic

[SkywingvL]

I'm announcing the release of the Advanced Script Compiler for NWN2.  You can download it here.  The NWVault page for the project can be found here.  (Old, defunct vault page: old page.)

The Advanced Script Compiler is an improved NWScript compiler, both for the toolset (as a toolset plugin), and for standalone usage (as a console application suitable for batch compilation).  It is based on Edward T. Smith (Torlack)'s nwnnsscomp, but with numerous bugfixes and improvements, plus updating to work seamlessly with NWN2 versus only being able to load script files from NWN1 KEY/BIFs as with nwnnsscomp.

The toolset plugin version brings several improvements over the stock toolset script compiler.  You are no longer limited to just seeing the first compile error (any beyond the first go to the verify window), dangerous constructs now produce a compiler warning versus being allowed silently, and the script code produced can be optimized in some cases.  Additionally, most compiler error/warning diagnostic messages are generally more informative than the stock compiler errors.

Source will be released with the next NWN2 Datafile Accessor Library refresh.

Installation instructions for how to install the Advanced Script Compiler as a toolset plugin are included, as well as the standalone version.  Once installed as a toolset plugin, script compilation seamlessly uses the new compiler.

Since the NWScript compiler started by ETS seems to be more or less without a maintainer, I'm taking up maintainership for it.

----

Release notes for initial release (also included in the readme for the standalone compiler in the download package):

Improvements over the stock nwnnsscomp include:

- New -l option to load resources from the game zip files.

- New -b option for true batch mode under Windows.

- Nested structs are now usable and do not produce an erroneous syntax error.

- Case statement blocks now obey proper rules for variable declarations; it is
  no longer possible to declare a variable that is skipped by a case statement,
  which would not compile with the standard (toolset) script compiler.

- The main and StartingConditional symbols are now required to be functions.

- Prototypes that do not match their function declarations are now fixed up by
  the compiler (with a warning) if compiler version 1.69 or lower is specified.
  This allows several broken include scripts shipped with the game to compile
  Note that a prototype cannot be fixed up if a function is called before the
  real declaration is processed.  In this case, an error is generated; the
  standard compiler generates bad code in such a case.  A prototype that is
  fixed up by the compiler generates a compiler warning.

- Uninitialized constants (const int x;) now generate a warning instead of an
  error if compiler version 1.69 or lower is specified.  The standard script
  compiler silently permits such constructs.

- An error is generated if a string literal's length exceeds 511 characters and
  compiler version 1.69 or lower is specified.  This is consistent with the
  standard script compiler's internal limitations.

- Attempts to use a switch statement while immediately located within a
  do/while block now generate a warning.  In the standard script compiler, such
  constructs cause bad code generation due to a compiler bug.

- Nested structure element access now generates a warning.  In the standard
  script compiler, a compiler bug may result in compilation errors or bad code
  generation with nested structure usage.

- Modifying a constant global variable before it is declared now generates a
  warning.  In nwnnsscomp, such constructs could either cause an internal
  compilert assertion failure, or could result in bad code generation.

- Script disassembly now generates a high level IR output (.ir), and a high
  level optimized IR output (.ir-opt).

- Scripts can now be loaded directly from a module.

- NWN1 game resource directories no longer are required to be present on the
  Windows build.

- NWN1-style resource directories can be used with the -1 command line option.

Edited by SkywingvL, 29 June 2014 - 01:00 AM.

[Spellfizzle]

Very nice!

[Morbane]

The external version, when I run it - it just flashes a window that then disappears instantly - is this a bug or am I doing something wrong?

[painofdungeoneternal]

The external one is not used by just running it as is, you can integrate it with a coding environment, visual studio, batch files ( like the make system on unix or for automated assembly after getting files from a svn repository), or eclipse, or just a better text editor. Integration requires understanding the parameters.

Regardless it is more akin to a dos command line tool which just compiles things, you should just use the toolset plugin unless there is something else you want to use.

Edited by painofdungeoneternal, 07 May 2011 - 09:46 PM.

[Morbane]

Thanks Pain - I do already use Programmers Notepad with nssclc for external use. I was just curious about this new tool and the modifications/improvements over the toolset compiler (which PN2 w nssclc uses). Truthfully those changes mentioned above cover things I'll probably never use in NWN2. Just the same I am still curious so maybe I'll look at the plug in version.

[SkywingvL]

Morbane wrote...

The external version, when I run it - it just flashes a window that then disappears instantly - is this a bug or am I doing something wrong?

It's a console application.  Try running it from cmd.exe if you want to see the output.

Edited by SkywingvL, 08 May 2011 - 12:36 AM.

[Morbane]

Thats cool - the first page has a list of flags etc. Now I am going to look at it within the toolset...

[SkywingvL]

Morbane wrote...

Thats cool - the first page has a list of flags etc. Now I am going to look at it within the toolset...

The toolset plugin version, if you're curious, always essentially behaves as though -c -v1.69 -e -l -m <current module> were in use.

GUI, conditional, and parameterized scripts are double checked against the stock compiler to ensure they'll work at runtime, as the stock compiler must be able to parse those scripts in the context of nwn2server.  (However the improvements I've made to Torlack's compiler bring it nearly completely inline with the stock compiler, so it's very rare that this may find something.  The only known conformance issue is matching the behavior of filling up the internal identifier list of the stock compiler.)

Edited by SkywingvL, 08 May 2011 - 03:12 AM.

[Bonzoli]

Works great, been using the previous one for a few weeks now as a plugin for nwn2. Just used this one, it also worked. The improved errors help quite a bit debuging scripts.

Thanks Skywing!

[SkywingvL]

Source now available with the public_nwn2dev refresh I just put out: here.

Edited by SkywingvL, 08 May 2011 - 05:02 AM.

[Morbane]

The plugin version works well with the PowerBar plugin - no issues so far - so auto parameters plus improved error checking together - nice thing that.

[kevL]

Do these look like good parameters for general usage :

-glo -h "C:\\<MyDocsPath>\\Neverwinter Nights 2" -n "F:\\Neverwinter Nights 2" *.nss


or should i have the [-e -v1.69] in there?

thanks, Sky.

[SkywingvL]

I would use -v1.69 too; this ensures conformance with the baseline toolset compiler by forbidding constructs that cause problems with BioWare's compiler.

If -h is not specified, <userprofile-dir>\\Neverwinter Nights 2 is assumed.
If -n is not specified, then the path in the registry for the NWN2 install (if it's present -- requires that you actually ran the NWN2 installer on this machine) is assumed.

If you're going to run the command line compiler repeatedly, you can improve performance by either:

1) Extracting the include scripts your scripts reference from the game zips, and passing that location as an include path with -i, and not specifying -l (load resources from game datafiles).  This avoids having to unzip the files every time internally.  At a minimum, you'd need to extract nwscript.nss before you can do away with -l.
2) Compiling all of your scripts at once with the same nwnscriptcompiler invocation if you prefer to use -l, which avoids having to re-read includes from the game zips each time.

Edited by SkywingvL, 13 May 2011 - 06:06 PM.

[kevL]

k, excellent. It's already running significantly quicker than my old setup (about 2 seconds w/ above -line and a Pentium D 925 3gHz (from inside an editor)).

just gotta say it: Thanks again, Skywing (a bit more testing and it'll be set)

[SkywingvL]

I've posted release 2 of the script compiler. This release includes some bugfixes for the toolset plugin only. No changes are made to the standalone compiler.

The same download link will work.

Changelog:

Release 2:
----------

- Fixed a stability issue with GUI and conditional script detection in the
toolset compiler plugin. This does not impact the standalone compiler.

- Fixed a resource prioritization issue with the toolset compiler plugin that
might result in resource files from the game data zips taking priority over
in-module files (for example, overridden scripts with the same name in the
module might not be used as expected in lieu of the in-zip versions). This
does not impact the standalone compiler.

[SkywingvL]

I've posted release 3 of the script compiler. This release includes some UI improvements for the toolset plugin (in particular, the several-times-requested support for clicking a compiler error or warning in the verify window to jump to that file and line -- even works for scripts in HAKs).  A bugfix for the standalone compiler is also included.

The same download link will work.

Changelog:

Release 3:
----------

- The toolset compiler plugin now scrolls to the first compiler error or
warning message in the verify window when compiling a script.

- Compiler errors and warnings in the verify output window are now clickable
when using the toolset compiler plugin.

- Disassembly mode (-d) for the standalone compiler was not producing IR
disassembly files (*.ir, *.ir-opt) properly. This has been fixed.

Edited by SkywingvL, 20 May 2011 - 06:36 AM.

[SkywingvL]

Release 4 posted with a number of core compiler improvements that improve conformance with bugs in the stock BioWare compiler in edge cases, as well as new compiler diagnostic warnings/errors for additional constructs that do not work correctly in NWScript (such as certain dangerous classes of global variable usage). This release updates both the standalone compiler and the toolset plugin.

Changelog:

Release 4:
----------

- In BioWare-compatible mode (enabled by default in the toolset plugin, and
enabled with -v1.69 (or lower) in the standalone compiler), an error
diagnostic is issued if a function with more than 32 arguments is declared,
as such functions are not supported in the standard compiler. In extensions
mode, a warning diagnostic is issued in this case.

- The standalone compiler's support for batch mode (-b argument) did not
support source file names that were wildcards correctly. This has been
corrected.

- The standalone compiler now supports supplying command line arguments via a
response file using the @ResponseFile argument. Only one response file is
permitted for the entire argument set, and the response file should be the
last argument on the command line as response file arguments are currently
always parsed after normal command line arguments.

- STORE_STATE references generated before main (i.e. within #globals) no longer
result in an uninitialized value being specified as the local variable store
count to the OP_STORE_STATE instruction.

- The compiler now issues a warning diagnostic should script code attempt to
directly invoke an engine function taking an "action"-typed arguments from a
global variable initializer expression. This case is not supported and will
cause an error at script execution time in most instances.

- The compiler now issues a warning diagnostic should script code attempt to
invoke a function which makes references to global variables (anywhere within
its call tree), from within a global variable initializer expression. This
case is not supported and will cause an error at script execution time in
most instances.

- The compiler now issues a warning diagnostic in compatible mode (-v1.69 or
below) if a script program has a function or global scope where the sum
total of all non-constant global variables and all local variables within
the current function (if not at global scope) exceed 1024. The
stock/standard compiler cannot compile such scripts due to a bug.

- The compiler now issues an error diagnostic if a local variable is declared
with the "const" qualifier, instead of triggering an assertion failure. This
construct causes a compile error with the stock compiler.

- The error diagnostics for misuse of the "const" qualifier have been improved
to provide more context information (i.e., specify which identifier had the
problem).

- The compiler now issues a warning diagnostic in compatible mode (-v1.69 or
  below) if a script program exceeds 16384 global identifiers.  This conforms
  with an internal limitation present in the stock compiler.

Edited by SkywingvL, 01 June 2011 - 09:54 AM.

[SkywingvL]

Release 5 posted with some improvements for both the toolset and standalone compilers. If you are upgrading a toolset compiler plugin installation, please be sure to copy over all of the required files in the toolset directory in the package.

--

Release 5:
----------

- The compiler now issues a warning diagnostic if the user attempts to declare
default arguments to an entry point symbol (main, StartingConditional). The
script runtime always ignores the user's supplied default arguments, so it
is misleading to declare defaults in the script.

- The compiler warning diagnostic for non-default arguments following default
arguments has been improved (i.e. specify which function and argument had the
problem).

- The standalone compiler was not properly loading KEY/BIF files if -1 was
supplied on the command line for NWN1 compatible mode. This has been fixed.

- IR disassembly was showing reversed operands for DIVFI instructions. This
has been fixed.

- User include directories were being prioritized after the resource system and
not before. This has been fixed. User include directories are now the
highest search priority for locating include files.

- The compiler would generate an assertion failure if one attempted to compile
a script whose entry point symbol was explicitly called within the script
itself. This has been fixed.

- The IR disassembler would reject as malformed (due to a mismatched stack
pointer across control flows) a script program whose entry point symbol took
parameters and was explicitly called within the script itself. This has been
fixed.

- The compiler now issues an error diagnostic should an identifier be used with
multiple incompatible symbol types (i.e. as a variable and a structure, or
other symbol type combinations).

- The compiler now allows forward declaration of structures, but only if the
structure in question is not embedded within another structure before it is
fully declared.

- The compiler now issues an error diagnostic should a structure be declared
twice.

- The compiler now longer issues an assertion failure in cases where a variable
has the same name as an existing symbol, such as a function or structure.

- The standalone compiler can now be directed to load a module from an
arbitrary path with the -r <path> option.

[SkywingvL]

Release 6 posted, with some new extensions for extensions mode in the command line compiler, and some diagnostic (error/warning) reporting improvements for the command line compiler and the toolset plugin.  Be sure to copy the new *.ndl file over if you are upgrading a toolset plugin install.

--

Release 6:
----------

- The compiler now correctly issues an error diagnostic if a script program
attempts to call a function that has been declared but never defined, instead
of raising an internal compiler assertion.

- The compiler provides support for 'default functions' in extensions mode,
if -e is supplied on the compiler command line. A default function can have
no body and if called, the compiler will automatically substitute a default
implementation that throws the arguments away and returns the appropriate
default value for any return value(s) as appropriate. This feature can be
activated via a #pragma default_function(function-identifier) directive after
a prototype for "function-identifier" is observed, but before the function is
defined. If a default function is defined as normal, the script program's
explicit definition is used as the function body instead of the default
definition.

- The compiler now supports automatic analysis and verification of scripts
as a compilation post-step in order to check for various inconsistencies.
Script analysis and verification can be enabled via the -a command line
option at a modest (10-20%) cost to compile time on most scripts.

- The compiler supports __FILE__, __LINE__, __COUNTER__, __FUNCTION__,
__NSC_COMPILER_DATE__, and __NSC_COMPILER_TIME__ predefined macros. The
macros work the same as their typical C counterparts (with the compiler
macros reflecting the build time for the compiler itself). These macros are
only enabled in extensions mode.

- The compiler now supports #error and #warning to issue diagnostics to the
programmer. These directives are only enabled in extensions mode.

- The compiler now has limited #if/#elif/#ifdef/#ifndef/#else/#endif support.
Only literal integers and macros can currently be tested; there is no support
for expressions on preprocessor #ifs constructs. The preprocessor #if
constructs are only enabled in extensions mode.

- The compiler no longer generates bad code when optimizations are enabled, a
global variable is initialized with a function call expression, and the
global variable in question is never written to after assignment.

- In compatibility mode, a warning is now issued if a script program contains a
for statement with an initializer or increment block whose type is not an
integer.

- Compiler error and warning messages now have easily searchable identifiers
associated with them (for example, NSC1001 or NSC6018).

Edited by SkywingvL, 15 September 2011 - 06:55 AM.

[kevL]

Tony the Tiger says, Great!!!

[rjshae]

Very cool enhancements. Thanks for posting this.

[SkywingvL]

Release 7 of the NWNScriptCompiler is available. You can grab the updated compiler module here: http://nwvault.ign.c...ns.Detail&id=99

Be sure to update NWNScriptCompilerDll.ndl when using the toolset plugin.

---

Changelog:

Release 7:
----------

- Script programs that contain duplicate definitions for a function, and which
supply empty bodies for such duplicate definitions now correctly trigger a
function redefinition error (NSC1062). Previously, these were incorrectly
permitted (the standard compiler does not permit such constructs).

- Error and warning diagnostic messages that relate to duplicate identifier
definitions now contain a source location pointer to the first definition of
the conflicting identifier.

- Fixed a bug where the command line compiler did not report the number of
errors across all scripts correctly in some circumstances. The error
message listing the total error count was showing an incorrect error count.

[SkywingvL]

Release 8 posted. This release includes updated toolset and standalone compilers. When upgrading the toolset compiler, be sure to upgrade ALL of the compiler DLLs.

Release 8:
----------

- Fixed warning message NSC6019 sometimes showing an incorrect count for the
standard compiler's internal identifier limit.

- IR disassembly output (.ir and .ir-opt files in NWNScriptCompiler -d mode)
now includes the literal values of constant variables in the disassembly
text.

- The compiler no longer generates a second NSC6018 (unsupported pragma
directive ignored) for each line causing the warning to appear.

- Added support for enabling full extensions mode in the toolset compiler,
which requires the NWScript Accelerator plugin to be used server-side for
parameterized scripts to function.

- New warning NSC6021 issued in compatibility mode when a nested assigment
where the RHS expression being assigned is itself an assignment, e.g.
"a = b = c;", is encountered. This does not parse properly in the
standard compiler.

You can find more information about the NWScript Accelerator plugin in the
following thread:

http://www.nwnx.org/...opic.php?t=1803

Extensions mode features include #ifdef, #define, #if, nested structures,
and other enhancements. See the standalone compiler's readme for full
details.

[SkywingvL]

Re-released this version to fix a bug where some modules were not being loaded properly in the toolset if the plugin was installed.

[kevL]

*bump*

This is still my most used app/plugin