Advanced Script Compiler for NWN1 release thread
#26
Posté 09 août 2011 - 07:31
Thanks,
Funky
#27
Posté 09 août 2011 - 10:35
I wouldnt mind having a go at it with c# - if the compiler functionality is in a dll, I could make a wrapper dll that would allow the threaded compiling.
My nickname at work is 'The Thread Wizard'
Basically, the way I would do it would be a
For each string s in ListOfNSS
{
if(ThreadCount >= Max_Threads)
Thread.Sleep(1000) // Keeps us at a manageable level of thread usage.
CompilerFunction(s)
}
Of course, I have no idea if its at all possible to make a c# equiv of this.
#28
Posté 12 août 2011 - 06:02
If you are going to perform active work on it then I will need to refresh the drop with a more current copy.
There are various components in the compiler tree that are known to *not* be thread-safe and will require modifications to the compiler core in C++ before the compiler can safely be used in the manner that you describe: the resource loading system plus various compiler internals, primarily in the parser handlers.
Previously, I did some work to eliminate most of the global variable usage in the compiler core itself (relative to Torlack's tree), other than the old bison-style parsers that really do need a global to work as they don't otherwise have any way to pass a context pointer around. The work I did is enough to allow you to safely create multiple compiler contexts but not use them concurrently from multiple threads.
Those globals could be turned into thread-local storage a-la TlsAlloc/TlsGetValue/TlsSetValue but there remains the problem of how to access the resource system which is only designed to be called from a single thread. Be careful of using __declspec(thread) as the wrapper DLL will not work properly if you use that unless the host OS is Vista or later due to a lack of OS support in old OS releases for __declspec(thread) in runtime-loaded DLLs (after process startup).
The compiler wrapper DLL (which NWNTX calls) also needs some locking added if support for concurrent compilation is added. Any reasonable approach to concurrent compilation with the existing code base is going to involve creating a compiler context per thread instead of trying to reuse the same context for multiple threads.
The NWN2 Toolset Plugin, included with the public_nwn2dev drop, illustrates how to call the compiler wrapper DLL from managed code (C#). Be aware that the NWN1 toolset is entirely native code.
Modifié par SkywingvL, 12 août 2011 - 06:07 .
#29
Posté 12 août 2011 - 06:10
I don't believe the old compiler ever supported parallel compilation natively. There are a substantial number of fixes that would need to be made to the compiler tree to make that work. Certainly the PRC build did not do this.FunkySwerve wrote...
Something we've noticed while using this compiler is that it no longer seems to have the capability of parallel compiling, which the old one did. This makes it significantly slower, in the end - it compiles in about 3-4 minutes what took around 110 seconds before. Is there any way this capability could be readded?
Thanks,
Funky
You could, however, spawn multiple NWNScriptCompiler processes and divy the work up between them as part of your build system. (This is not entirely free: you will incur heavier I/O overhead as a result of having to (re)load the same resources multiple times. But you would still get better performance if you're CPU bound.)
I would investigate integrating parallel builds via multiple compiler processes into your build system as the most realistic approach here. Since typical C/C++ compilers are also not parallelized, it is not uncommon for build systems to have some support for spawning multiple compiler processes to split the work up over multiple processors.
Modifié par SkywingvL, 12 août 2011 - 06:16 .
#30
Posté 15 août 2011 - 03:30
My own investigations into doing the multi-threaded compiling went bust.
I would be interested in seeing someone releasing a prototype just to see if it is feasible.
Eg - For each nss in ListOfNSS
{
while(CURRENT_THREADS >= MAX_THREADS) //Prevents us from going mad
{
Thread.Sleep(100);
}
DoCompile(nss);
}
This is how I would visualize it in C#, of course, I have no idea how to impliment this in a C++ Project.
#31
Posté 15 août 2011 - 03:54
Funky
#32
Posté 15 août 2011 - 11:21
FunkySwerve wrote...
This (spawning multiple processes) is exactly what we were doing with the PRC compiler. We can't seem to make it work with the new build. More as it comes...
Funky
There should be no issue with running multiple instances simultaneously; I would be curious to understand what error conditions you're running into here.
#33
Posté 15 août 2011 - 11:23
Baaleos wrote...
Well,
My own investigations into doing the multi-threaded compiling went bust.
I would be interested in seeing someone releasing a prototype just to see if it is feasible.Eg - For each nss in ListOfNSS { while(CURRENT_THREADS >= MAX_THREADS) //Prevents us from going mad { Thread.Sleep(100); } DoCompile(nss); }This is how I would visualize it in C#, of course, I have no idea how to impliment this in a C++ Project.
Baaleos, this assumes that the underlying compiler logic is thread-safe (it isn't). You would need to rework the parts of the compiler that aren't thread safe to either acquire brief synchronization (best choice for the resource management logic) or use thread local storage (e.g. the various handler routines called by the bison generated parser).
An alternative option would be to run the existing code in multiple processes and provide some multiplexing mechanism to bridge the resource system in the toolset to those processes.
#34
Posté 20 août 2011 - 02:04
i get: "WARNING: Failed to open .key archive 'C:/NeverwinterNights/NWN//xp1patch.key': exception 'Failed to open KEY file.'."
(It is adding 2 "/" intead of 1)
Modifié par Mavrixio, 20 août 2011 - 02:04 .
#35
Posté 20 août 2011 - 05:52
Funky
#36
Posté 20 août 2011 - 09:45
[It's not a fatal error if a particular .key file doesn't exist so long as the resources that scripts ask for can still be found. The compiler will still look for the other .key files in its list.]
Modifié par SkywingvL, 20 août 2011 - 09:47 .
#37
Posté 20 août 2011 - 10:13
Is it still possible to remove this warning, if the parameter -1 is set? I would like to run the compiler many time for different group of scripts and then ouput all in a log file with the silent mode and get only the error in it, currently it get filled by this warning.
Modifié par Mavrixio, 20 août 2011 - 10:14 .
#38
Posté 21 août 2011 - 01:20
I could remove the warning entirely.
#39
Posté 21 août 2011 - 01:42
#40
Posté 24 août 2011 - 12:38
#41
Posté 24 août 2011 - 12:53
#42
Posté 01 septembre 2011 - 04:44
#43
Posté 03 septembre 2011 - 05:18
ditto :-)
<...to lurk>
#44
Posté 15 septembre 2011 - 06:57
--
Release 6:
----------
- The compiler now correctly issues an error diagnostic if a script program
attempts to call a function that has been declared but never defined, instead
of raising an internal compiler assertion.
- The compiler provides support for 'default functions' in extensions mode,
if -e is supplied on the compiler command line. A default function can have
no body and if called, the compiler will automatically substitute a default
implementation that throws the arguments away and returns the appropriate
default value for any return value(s) as appropriate. This feature can be
activated via a #pragma default_function(function-identifier) directive after
a prototype for "function-identifier" is observed, but before the function is
defined. If a default function is defined as normal, the script program's
explicit definition is used as the function body instead of the default
definition.
- The compiler now supports automatic analysis and verification of scripts
as a compilation post-step in order to check for various inconsistencies.
Script analysis and verification can be enabled via the -a command line
option at a modest (10-20%) cost to compile time on most scripts.
- The compiler supports __FILE__, __LINE__, __COUNTER__, __FUNCTION__,
__NSC_COMPILER_DATE__, and __NSC_COMPILER_TIME__ predefined macros. The
macros work the same as their typical C counterparts (with the compiler
macros reflecting the build time for the compiler itself). These macros are
only enabled in extensions mode.
- The compiler now supports #error and #warning to issue diagnostics to the
programmer. These directives are only enabled in extensions mode.
- The compiler now has limited #if/#elif/#ifdef/#ifndef/#else/#endif support.
Only literal integers and macros can currently be tested; there is no support
for expressions on preprocessor #ifs constructs. The preprocessor #if
constructs are only enabled in extensions mode.
- The compiler no longer generates bad code when optimizations are enabled, a
global variable is initialized with a function call expression, and the
global variable in question is never written to after assignment.
- In compatibility mode, a warning is now issued if a script program contains a
for statement with an initializer or increment block whose type is not an
integer.
- Compiler error and warning messages now have easily searchable identifiers
associated with them (for example, NSC1001 or NSC6018).
#45
Posté 20 septembre 2011 - 12:36
Thanks for the update!
I installed it on our development system and have a few more requests concerning the output.
Here's one of the command line that i use:
So yea, its on linux with wine which may explain the weird bug that i get bellow.wine /home/sinfar/nwn/../nwn_tools/NWNScriptCompiler.exe -1cegloyq -n /home/sinfar/nwn -i '/home/sinfar/nwn/modules/temp/tests3On09;/home/sinfar/nwn/modules/resources/agl_;/home/sinfar/nwn/modules/resources/agn_;/home/sinfar/nwn/modules/resources/apt_;/home/sinfar/nwn/modules/resources/aquatic_;/home/sinfar/nwn/modules/resources/banker_;/home/sinfar/nwn/modules/resources/barmaid_;/home/sinfar/nwn/modules/resources/bdsm_;/home/sinfar/nwn/modules/resources/bequip_;/home/sinfar/nwn/modules/resources/bi_;/home/sinfar/nwn/modules/resources/bindtl_;/home/sinfar/nwn/modules/resources/bkstore_;/home/sinfar/nwn/modules/resources/bstone_;/home/sinfar/nwn/modules/resources/cac_;/home/sinfar/nwn/modules/resources/cchall_;/home/sinfar/nwn/modules/resources/chess_;/home/sinfar/nwn/modules/resources/chloe_;/home/sinfar/nwn/modules/resources/cmbtz_;/home/sinfar/nwn/modules/resources/craft_;/home/sinfar/nwn/modules/resources/ctf_;/home/sinfar/nwn/modules/resources/deadz_;/home/sinfar/nwn/modules/resources/dm_;/home/sinfar/nwn/modules/resources/dms_;/home/sinfar/nwn/modules/resources/dmsl_;/home/sinfar/nwn/modules/resources/drink_;/home/sinfar/nwn/modules/resources/drow_;/home/sinfar/nwn/modules/resources/ee_;/home/sinfar/nwn/modules/resources/elc_;/home/sinfar/nwn/modules/resources/ele_;/home/sinfar/nwn/modules/resources/emo_;/home/sinfar/nwn/modules/resources/fa_;/home/sinfar/nwn/modules/resources/feternal_;/home/sinfar/nwn/modules/resources/g_;/home/sinfar/nwn/modules/resources/gmbl_;/home/sinfar/nwn/modules/resources/hench_;/home/sinfar/nwn/modules/resources/holo_;/home/sinfar/nwn/modules/resources/home_;/home/sinfar/nwn/modules/resources/hub_;/home/sinfar/nwn/modules/resources/icf_;/home/sinfar/nwn/modules/resources/inc_;/home/sinfar/nwn/modules/resources/jdj_;/home/sinfar/nwn/modules/resources/jewel_;/home/sinfar/nwn/modules/resources/jj_;/home/sinfar/nwn/modules/resources/joust_;/home/sinfar/nwn/modules/resources/kaala_;/home/sinfar/nwn/modules/resources/lil_;/home/sinfar/nwn/modules/resources/lu_;/home/sinfar/nwn/modules/resources/lwjb_;/home/sinfar/nwn/modules/resources/lwpt_;/home/sinfar/nwn/modules/resources/magi_;/home/sinfar/nwn/modules/resources/mappin_;/home/sinfar/nwn/modules/resources/mbm_;/home/sinfar/nwn/modules/resources/mcloth_;/home/sinfar/nwn/modules/resources/mforge_;/home/sinfar/nwn/modules/resources/mkey_;/home/sinfar/nwn/modules/resources/mkpc_;/home/sinfar/nwn/modules/resources/mm_;/home/sinfar/nwn/modules/resources/mmat_;/home/sinfar/nwn/modules/resources/mmm_;/home/sinfar/nwn/modules/resources/mob_;/home/sinfar/nwn/modules/resources/mod_;/home/sinfar/nwn/modules/resources/mons_;/home/sinfar/nwn/modules/resources/msds_;/home/sinfar/nwn/modules/resources/mtm_;/home/sinfar/nwn/modules/resources/mtss_;/home/sinfar/nwn/modules/resources/mystc_;/home/sinfar/nwn/modules/resources/nboard_;/home/sinfar/nwn/modules/resources/nurse_;/home/sinfar/nwn/modules/resources/nw_;/home/sinfar/nwn/modules/resources/ooc_;/home/sinfar/nwn/modules/resources/oz_;/home/sinfar/nwn/modules/resources/pc_;/home/sinfar/nwn/modules/resources/pcman_;/home/sinfar/nwn/modules/resources/pcplc_;/home/sinfar/nwn/modules/resources/ph_;/home/sinfar/nwn/modules/resources/pj_;/home/sinfar/nwn/modules/resources/plcvfx_;/home/sinfar/nwn/modules/resources/possess_;/home/sinfar/nwn/modules/resources/pvpisld_;/home/sinfar/nwn/modules/resources/pwstore_;/home/sinfar/nwn/modules/resources/pzl_;/home/sinfar/nwn/modules/resources/rentrm_;/home/sinfar/nwn/modules/resources/rp_;/home/sinfar/nwn/modules/resources/s_;/home/sinfar/nwn/modules/resources/sail_;/home/sinfar/nwn/modules/resources/sc_;/home/sinfar/nwn/modules/resources/scroll_;/home/sinfar/nwn/modules/resources/sex_;/home/sinfar/nwn/modules/resources/sge_;/home/sinfar/nwn/modules/resources/shckle_;/home/sinfar/nwn/modules/resources/sinfar_;/home/sinfar/nwn/modules/resources/sinstab_;/home/sinfar/nwn/modules/resources/skl_;/home/sinfar/nwn/modules/resources/social_;/home/sinfar/nwn/modules/resources/ssm_;/home/sinfar/nwn/modules/resources/sss_;/home/sinfar/nwn/modules/resources/sstrip_;/home/sinfar/nwn/modules/resources/ssv_;/home/sinfar/nwn/modules/resources/tl_;/home/sinfar/nwn/modules/resources/tnb_;/home/sinfar/nwn/modules/resources/toys_;/home/sinfar/nwn/modules/resources/tpetshp_;/home/sinfar/nwn/modules/resources/tpvp_;/home/sinfar/nwn/modules/resources/txtedit_;/home/sinfar/nwn/modules/resources/vc_;/home/sinfar/nwn/modules/resources/vg_;/home/sinfar/nwn/modules/resources/wastes_;/home/sinfar/nwn/modules/resources/wbook_;/home/sinfar/nwn/modules/resources/welcm_;/home/sinfar/nwn/modules/resources/wstc_;/home/sinfar/nwn/modules/resources/wz_;/home/sinfar/nwn/modules/resources/xs_;/home/sinfar/nwn/modules/resources/yasm_;/home/sinfar/nwn/modules/resources/zep_;/home/sinfar/nwn/modules/resources/zz_' "/home/sinfar/nwn/modules/temp/tests3On09/*.nss" >> /tmp/test_changelogLIlF0U
This is the output:
- The number of errors seem wrong.test_debug.nss(8): Error: NSC1040: Syntax error at "}"
Compilation aborted with errors.
Error: Failed to process file "/home/sinfar/nwn/modules/temp/testrjqrHM/test_debug.nss".
test_disturb.nss(15): Error: NSC1040: Syntax error at "SendMessageToPC"
Compilation aborted with errors.
Error: Failed to process file "/home/sinfar/nwn/modules/temp/testrjqrHM/test_disturb.nss".
3339736249 error(s); see above for context.
- Use one new line instead of 2.
- Output the "Compilation aborted with errors." and "3339736249 error(s); see above for context." only in normal mode but not in quiet mode as those can be interpreted.
Other than that, the script errors are volontary and all the scripts get compiled correctly.
Modifié par Mavrixio, 20 septembre 2011 - 12:39 .
#46
Posté 18 mai 2012 - 10:17
I was very interested in this new script compiler for continuing development in NWN1. I have a lot of scripts I wrote for the Bard Tale 1 module rebuild I did in the vault some years ago, and I've been now tinkering with the Bard's Tale 2: The Destiny Knight module(s) when I stumbled across this great new tool.
But for some reason it crashes when compiling the scripts I've written in the past.
To get straight to the point, I have a very simple script that crashes the compiler without much ceremony. I'll include it here as it's pretty small. Perhaps you can let me know what's wrong with it.
#include "NW_I0_GENERIC"
int StartingConditional()
{
return (GetAssociateState( NW_ASC_DISARM_TRAPS ) == FALSE) ? TRUE : FALSE;
}
The command line I'm using is:
NWNScriptCompiler -1cl -r . -n G:\\NeverwinterNights\\nwn testscript.nss
Any help would be greatly appreciated.
-Agmorion
#47
Posté 20 mai 2012 - 09:45
#include "NW_I0_GENERIC"
int StartingConditional()
{
return (!GetAssociateState( NW_ASC_DISARM_TRAPS ));
}
it's a bit simplier and it will work exactly the same as your script.
#48
Posté 21 mai 2012 - 07:42
This doesn't repro with the NWN2 includes using the current compiler drop.
Can you link me a zip that includes all of the latest .nss files (including nwscript.nss) that ship with the current NWN1 patch? I don't think that I have the current version of these handy, and the repro might be dependant on something in the include set.
#49
Posté 24 mai 2012 - 03:40
http://www.nynaeve.n...iptCompiler.zip
Release 12:
-----------
- The pcode dump mode now works again for the standalone compiler.
- The compiler now allows compiled scripts to be up to 64MB, versus up to the
old limit of 512K. It is strongly recommended that scripts be kept under
512K compiled size for both compile-time and runtime performance reasons.
- The compiler now provides a proper compile time error (NSC1036) if a script
exceeded the maximum compiled script size. Previously, the compiler asserted
in this condition.
Release 11:
-----------
- In compatibility mode, the compiler will now terminate an unterminated
comment block at the end of an include file. This improves compatibility
with certain ill-formed third party scripts. In compiler version 1.70 or
higher, the fixup is not performed.
#50
Posté 26 mai 2012 - 10:37





Retour en haut







