20 réponses à ce sujet

[leo_x]

Thought I'd fork out a new topic for a Community Launcher / Auto-Downloader / Updater Project... Since it really hasn't got much to do with a gamespy replacement per se.

I've been working on a small prototype over the weekend, the following links would give a gist of what I think would be nice features to have:
README
configuration
template

Things like:
- Open Source. Implemented in a widely used, portable, language like Python.
- Simple text based file formats (JSON in this case).
- Easy PW specific dialog.tlk overrides.
- Preloading via nwnpatch.ini is in there. That plus the above could probably provide cross-platform resource preloading. I've not used nwnx_connect/NWNCX, so that might well be a much better solution.
- Auto-downloading / updating haks/tlks.
- Some support of resource clean-up (tho I've not bothered with that yet.)
- Hopefully easy enough to integrate with the new server listing web interfaces.

It's written in Python 2.7. It's currently basically a command line tool at this point, however it would be easy enough to put a GUI on top.

Anyone have an opinion on if there's enough need for such a thing and, if so, what features you'd like to see it have?

Modifié par pope_leo, 16 décembre 2012 - 01:31 .

[Rolo Kipp]

<now...>

Opinions? Got 'em =)
First: Yes! *cough* Let me say that another way... YES! <subtle>
(wizard's are, you know) <i was being sarcaastic>
(I wasn't)

Second: I would like to suggest something built around Ruby and the nwn-lib (Thanks to Funky & Virusman for pointing me in that direction!). It handles all the nwn file formats and could (eventually) provide a true package manager functionality with dependencies and conditionals handled neatly.

On the other end of it, cataloging and indexing content, we'd need a resource repository... something big and strong, like, oh, I don't know, a Vault ;-)

<...you've done it>

[Shadooow]

perfect, question is, since player must download this already, why not to implement also nwncx and other (custom nwn clients such as HGLL client or the one I have from PvN) features?

- master server connection bypassed
- camera unlocked (possibly optional via ini setting)
- base classes unlocked
- custom weapon VFXes
- custom beams
- etc...

[Rolo Kipp]

<in perfect...>

Integrating nwnx & nwncx are actually one of my personal goals, since I think the future of NwN really requires the power of NwNx.

The custom clients would be quite valid targets for a package manager.

<...or nearly perfect, agreement>

[T0r0]

This excites me.

[leo_x]

Rolo Kipp wrote...
Second: I would like to suggest something built around Ruby and the nwn-lib (Thanks to Funky & Virusman for pointing me in that direction!). It handles all the nwn file formats and could (eventually) provide a true package manager functionality with dependencies and conditionals handled neatly.

Ruby would fit the profile for a good implimentation language.  It used to be that Windows wasn't exactly a first-class platform, but I think that's changed.  It would be worth forming around the same language.  

Edit:
If I'm reading the thread you linked to you'd want a package manager on the individual asset level?  I mean, what would be the level of granularity you'd want to be able to control? Would haks be compiled on the client side?

I really like the idea so far as I think I understand you.  When I first started working on getting CC together for my PW (and I still haven't finished yet).  I wished I could just get a file for example: a new baseitem - the file would be called, say, shortquarterstaff.baseitem (a quarterstaff for gnomes and halflings, a toothpick for the others).  I put this file into a hak or folder and that's that, no 2das...  Maybe, grudgingly, modify a TLK.

Modifié par pope_leo, 17 décembre 2012 - 04:46 .

[leo_x]

ShaDoOoW wrote...

perfect, question is, since player must download this already, why not to implement also nwncx and other (custom nwn clients such as HGLL client or the one I have from PvN) features?

- master server connection bypassed
- camera unlocked (possibly optional via ini setting)
- base classes unlocked
- custom weapon VFXes
- custom beams
- etc...

Only to have some seperation of concerns.  Like Rolo said that stuff could easily be installed if the project expanded to more of a package manager project in general.

If that kind of functionality was included the project would probably need to have dependencies on C/C++ compilers, etc.  It's not clear if NWNCX stuff will ever be cross-platform.  Honestly, I'd doubt if anyone would even try to contribute to that project since Virusman is the only person with the client debugging symbols (and can't distribute them, so far as I've read) and slogging through the assembly to do that stuff would be insanely tedious for anyone else.

Modifié par pope_leo, 17 décembre 2012 - 09:30 .

[Rolo Kipp]

<like sand through the hour-glass...>

pope_leo wrote...
...
If I'm reading the thread you linked to you'd want a package manager on the individual asset level?  I mean, what would be the level of granularity you'd want to be able to control? Would haks be compiled on the client side?

I really like the idea so far as I think I understand you.  When I first started working on getting CC together for my PW (and I still haven't finished yet).  I wished I could just get a file for example: a new baseitem - the file would be called, say, shortquarterstaff.baseitem (a quarterstaff for gnomes and halflings, a toothpick for the others).  I put this file into a hak or folder and that's that, no 2das...  Maybe, grudgingly, modify a TLK.

*Exactly*! With version control and dependencies.

For instance. Tileset A is dependent on tile model A-1. Tile model A-1 uses texture A-1-A. If an upgrade to texture A1A is made, model A-1 is improved without actually editing the model. Tileset A is improved  without editing tileset. If tile model A-2 is later added to dependency, new tileset is generated.

And everything patched/built at run-time when the shell around nwnmain is "checking for updates" which includes checking for updates to the module being loaded and any dependencies.

The file formats are mostly understood. How to generate set files and gff files and erfs is understood. We know how a module is stored and how to parse it. We know how to build a hak. We know a lot of what we *need*.

Really I see only two things slowing us down: Coders willing to build the shell and gui and standards to make sure dependencies and revisions are implemented.

I.e. if I branch off a new "treegiant" to make my "Forrestal", modules looking for "Treegiant" will *not* find my "Forrestal". But if I improve the Treegiant, they will.

Or, a more complicated case, there are a *lot* of overrides out there that improve Bioware models and textures. Some would be considered improvements. Others would be branches into new resources. Many conflict, which must be resolved with branching the resource.

So, there'd be either a central or distributed repository for resources and the package manager would track what's needed on a *module* basis (a module.ini, anyone?) and would update/generate the *non* standard resources on the first-run/update available basis.

<...is the granularity *rolo* wants>

[Baaleos]

If this was in C# I would be able to assist - alas - im not familiar with ruby.

But if you are going to start adding in additional softwares, perhaps creating some sort of Gui screen with a list of checkboxes that indicate which softwares to be 'subscribed' to.

eg- NWNCX - When the launcher starts, it checks the version of the files, against the latest on the website, and auto updates
Also maybe give the option for some beta softwares out there like nwnshader?
(I mention it as a beta - cause I last time I used it, it was bit buggy)

Creating our own community launcher, does open up possibilities for massive customization of the client.
maybe even to the point of a community website - where the users are PW Hosts, and they can list the softwares they recommend for their PW's - and host / point to a download URL for their files?
Then the Downloader optionally downloads them too.

[leo_x]

Rolo Kipp wrote...
...
*Exactly*! With version control and dependencies.

It shouldn't be to hard to create an environment like you describe on top of one of the open source version control systems.  Some of them aren't great with binary formats tho...

Rolo Kipp wrote...
Really I see only two things slowing us down: Coders willing to build the shell and gui and standards to make sure dependencies and revisions are implemented.

Well, I'd be happy to work on this project with you.  I just downloaded Ruby and nwn-lib and am testing the waters...  I'm going to convert my launcher prototype to Ruby to get a better feel for it.

It sounds like there are a couple projects here that could be distinct, it might be best to tease them apart.  The launcher/downloader/updater (which needs to support the oldway of doing things), the module (de)composer, the CC repository (which itself can be broken into local and distributed subprojects) and the package manager to make them all work together.  Does that sound right to you?

Rolo Kipp wrote...
...
So, there'd be either a central or distributed repository for resources and the package manager would track what's needed on a *module* basis (a module.ini, anyone?) and would update/generate the *non* standard resources on the first-run/update available basis.

Could you expand on the non-standard part?  Do you mean custom content in general?  I'm not a CC creator, so you'll have to keep that in mind.

Modifié par pope_leo, 18 décembre 2012 - 11:39 .

[virusman]

I think something like this should interface with NWNCX if not implemented as a NWNCX plugin. Low-level hooks NWNCX is capable of providing may be useful.
I'll copy the NWNCX repository to github (along with other NWNX stuff) for easier access.

[Baaleos]

Hey Virusman - I was just wondering -
NWNCX - Would it be possible to hook/control the cinematic playing function in nwn client?

Myself and perhaps other server hosts have always wanted to be able to play video cinematics to their players.
but the only choice available, was the Intro or Outro for the module movie.

It would be great if NWNCX could receive net communications from nwnx - and then trigger cinematics etc on the client. - Which can of course be exited out of via the esc key.

[Rolo Kipp]

<tossing his chips...>

virusman wrote...
I think something like this should interface with NWNCX if not implemented as a NWNCX plugin. Low-level hooks NWNCX is capable of providing may be useful.
I'll copy the NWNCX repository to github (along with other NWNX stuff) for easier access.

Absolutely! That's kind of the point, to support the expansions and empowerments no longer possible through Bioware. We need to shift into 3rd-party support mode as completely as possible.

pope_leo wrote...
It shouldn't be to hard to create an environment like you describe on top of one of the open source version control systems.  Some of them aren't great with binary formats tho...

That's what I'm thinking, as well.

Well, I'd be happy to work on this project with you.  I just downloaded Ruby and nwn-lib and am testing the waters...  I'm going to convert my launcher prototype to Ruby to get a better feel for it.

It sounds like there are a couple projects here that could be distinct, it might be best to tease them apart.  The launcher/downloader/updater (which needs to support the oldway of doing things), the module (de)composer, the CC repository (which itself can be broken into local and distributed subprojects) and the package manager to make them all work together.  Does that sound right to you?

Yes, actually. In addition, we should start with rough granularity (on the hak/erf level) before working down to the individual resource level (my feeling - for backwards compatibility).

Could you expand on the non-standard part?  Do you mean custom content in general?  I'm not a CC creator, so you'll have to keep that in mind.

I mostly mean custom content. But here's the thing. We have the "standard" resources put out by Bioware that does not need to be distributed, and we have custom content that does. But we *also* have modifications to the standard resources (including code/script modifications) that need to be tracked and (optionally) applied to the standard resources *without* destroying the standard resources (have to be able to roll-back to initial state). So that's why I make that distinction.

For my part, I want to work on the repository end (begun (barely) at the nwn-ccc and the VPP).
I want to break every hak/erf/archive down to resources and wrap them in meta-data that preserves author/creation/edit date and their location on a resource tree (trunk - update, branch - modification, etc.). 
I.e. there are probably a hundred haks out there with Issig's hands. Most of them are identical. My extractor must be able to track them all down to the original and make that the parent node of the resource tree. This really means (at least on the repository end) I need MD5 hashes of every single resource. And more...

(In other words, we need to define metadata wrappers that will enable the package manager)

(More later, have to work :-P Gold, you know. The dwarf is crazy for it... *sigh*)

<...into the pot>

Modifié par Rolo Kipp, 18 décembre 2012 - 05:37 .

[Tyndrel]

Speaking for the majority of mere mortals that have only an average technical ability (or less), I understand enough of the above to know that this will be a huge improvement to our beloved game and provide facilities that we have only dreamed of until now. How this is achieved is of less importance to the story tellers and adventure crafters of the community, we can only look on in awe and hope that you good folks can provide your masterpiece with a user interface that is no more complex to use than the game itself. I shall be following this thread with baited breath though probably understanding only 25% of what is said. Thank you, from the heart of my bottom!

[Rolo Kipp]

<trying to make...>

@ T: Transparency is one of the primary goals. It just needs to work.

@ VM: NwNCx is simply the best candidate for the core of the Launcher.
The Launcher makes the modifications to the engine to enable enhanced services. It also needs to determine resources needed and construct the environment the engine (server and/or client) runs in. IOW, it's already doing some of what we need.

And low-level hooks are fantastic :-) "Communication is the key to a good relationship"! Always true.

OTOH, NwNx is better supported on linux (from what I read) than Windows and most of the installed *player* base is Win. NwNx really needs to be cross-platform and integrated into the server (again with the low-level hooks). We *need* strong db and internet functionality and 1.69 simply doesn't have it.

My personal opinion there is (and has been) that we need to deprecate the combined server/client completely and make a parallel server+client runtime environment standard. And transparent. No fiddling for Tyndrel ;-)

Time is short, so I'll just rephrase PL here:

Launcher - Wraps the engine, builds runtime environment (DLs/Updates as required) and launches module. Module oriented.
Repository(ies) - Stable storage of resources, *including* "plugins". resource oriented.
Manager - installs/manages setup and options for system. System oriented.
MOptimizer (Clinging stubbornly to that name!) - Module optimizer. Toolset oriented.

Gotta go!

<...all the pieces fit>

[leo_x]

Weeell... I'll toss in a little dissent, maybe concern would be a better word. This is no insult to NWNCX, it looks like a wonderful project. I think for a project this large to succeed it will need some... ruthless, for lack of a better word, commitment to clarity and maybe I'm missing what is meant.

1) I mentioned this above NWNCX is not currently cross-platform. Will it ever be? and with the level of features that it can/does/will have on Windows? With the Steam Linux client going into open beta and Valve announcing their "Steam console" (also built on Linux). I can imagine a world in which Linux is a much much more popular consumer platform (for games). A handful of people on my relatively small server use it already.

2) "The Launcher makes the modifications to the engine to enable enhanced services." Could you elaborate on what the enhanced services are? Is this what NWNCX already provides, VFX unlocking, etc? What it does today is pretty much totally orthogonal to a what I've taken 'launcher' to mean. 

3) "It also needs to determine resources needed and construct the environment the engine (server and/or client) runs in". Is the engine still consuming HAK files at this point? If by 'environment' you mean some client ResMan, sign me up to use it. But if it's HAKs slurped together with nwn-lib, I see no reason to either a) do all that in C++ or embed Ruby or c) call back out to a Ruby script/exe. It would be way simpler to pass command line arguments to NWNCX (maybe add a --dialog TLK switch for overriding the dialog.tlk) which it already passes on to nwmain after everything was downloaded/updated/constructed.  

Edit: I might be looking at this from a different perspective.  I just realized maybe you want to be able to use the client to navigate to a module / saved game / server in a gamespy list (if anyone replaces it) and then have all the 'magic' happen when it's loaded/connected to in game?

That's all I have time to write ATM.

Modifié par pope_leo, 20 décembre 2012 - 02:32 .

[Rolo Kipp]

<grunting...>

pope_leo wrote...
Weeell... I'll toss in a little dissent, maybe concern would be a better word. This is no insult to NWNCX, it looks like a wonderful project. I think for a project this large to succeed it will need some... ruthless, for lack of a better word, commitment to clarity and maybe I'm missing what is meant.

Clarity good! Communication good!

1) I mentioned this above NWNCX is not currently cross-platform. Will it ever be? and with the level of features that it can/does/will have on Windows? With the Steam Linux client going into open beta and Valve announcing their "Steam console" (also built on Linux). I can imagine a world in which Linux is a much much more popular consumer platform (for games). A handful of people on my relatively small server use it already.

Just to be clear, NwNCx is the client extender that Vm's been using to hook into the engine and provide client-side enhancements, the repository of which he is copying to github (along with NwNx stuff) for consideration. I.e. it is already something that works in the direction we are going.

I'm much more aware of the windows limitations in NwNx. I didn't realize NwNCx suffered similarly. But even so, it is a base to work from and the primary developer is familiar with ruby/nwn-lib. A big plus to me.

Anyway, that's the justification for my above statement. I think it's a viable candidate for Ascension (reading Steven Erickson :-P )

Ack! Rides here... more later

<...like a caveman in a hurry>

[Rolo Kipp]

<erasing...>

pope_leo wrote...
2) "The Launcher makes the modifications to the engine to enable enhanced services." Could you elaborate on what the enhanced services are? Is this what NWNCX already provides, VFX unlocking, etc? What it does today is pretty much totally orthogonal to a what I've taken 'launcher' to mean.

The Launcher, as I envision it, is a wrapper around the legacy engine that:
1) Enables enhanced services (VFX unlocking, skip master server, etc)
2) Loads module (metadata file?) and determines resources needed/missing. Collates resources into runtime environment for module.
3) Launches module on server (or server-client if people insist :-P) and connects client to server (if local, played in SP mode)

What are you picturing for the Launcher and what it needs to do?

3) "It also needs to determine resources needed and construct the environment the engine (server and/or client) runs in". Is the engine still consuming HAK files at this point? If by 'environment' you mean some client ResMan, sign me up to use it. But if it's HAKs slurped together with nwn-lib, I see no reason to either a) do all that in C++ or embed Ruby or c) call back out to a Ruby script/exe. It would be way simpler to pass command line arguments to NWNCX (maybe add a --dialog TLK switch for overriding the dialog.tlk) which it already passes on to nwmain after everything was downloaded/updated/constructed.  

The mechanics are TBD. I see, at least at the start, sticking with haks. Coarse granularity. 
But I *want* to move toward something like resman. And the reason for choosing the framework early is simply that it makes it easier when we really start getting tricky.

Edit: I might be looking at this from a different perspective.  I just realized maybe you want to be able to use the client to navigate to a module / saved game / server in a gamespy list (if anyone replaces it) and then have all the 'magic' happen when it's loaded/connected to in game?

Specifically, I want it to work as transparently as possible for the player/admin. Double-click on icon. NwN launches with menu. Pick module/server to play on. Launcher enables the environment NwN expects and then launches it.

Different services for clients and servers, btw. But the *Launcher* is *module* oriented and is all about getting the module loaded properly with service enhancements and optimized content (minimal resources).

<...the blackboard in a white cloud>

[leo_x]

Well, maybe we are on the same page, at least up to the enhanced services being in the launcher -- and my main issue with that is it isn't cross-platform.

I was using Launcher in the sense of what I proposed and prototyped in the OP. People could download and use it today probably. It's in Python tho. Here is the process assuming the player had the 'neverrun' launcher prototype installed:

1) Player finds server they want to try.
2) Server has Neverwinter Resource Locator (NRL) -- which is just a JSON file
3) Player dowloads said file. Double-clicking that file is essentially the sum total of user interface.
4) An example of what the system would generate for my server: here You'd need to look at the links in the OP to get a better idea of what it all meant.
5) They would double click on it. It would recognize that it's never been installed before. It would fetch whatever resources were needed to login.* Since most people already have picked a server would download the NRL mainly as a direct connector / updater, they'd probably not have to actually download anything.
6) Everything would be downloaded and installed.
7) Next time the player double-clicks the NRL it phones home (the source field in the JSON file described in the OP) and checks if it has changed. If not it direct connects you to the server. If so, it downloads and installs whatever is different and then direct connects you to the server.

The prototype does allow for a few modifications to nwmain.exe in order to facilitate a) server specific dialog.tlk overrides and preloading haks on a hak by hak basis** via nwnpatch.ini. This is the one place where NWNCX would be nice, but I feel that interfacing with it on the command line level would be nicer than being a plugin. I.e. in the launcher's configuration there could be a value saying "use_nwncx=1".  Then instead of modifying and running  'nwmain.exe <DIRECT CONNECT STUFF>' it could run 'nwncx.exe <DIRECT CONNECT STUFF> --dialog mydialog.tlk --preload a.hak b.hak ...'

I don't really care if that prototype goes to the dustbin, this is all to explain my perspective after implementing something that covers most of today's use cases (I think).

* On the first use of the launcher it generates sha1's (better than md5 from what I've read) for all modules, haks, tlks you have in your NWN directory.

** I'm not sure if nwnx_connect/NWNCX is like this, but at first I tried preloading everything and the shutdown time for NWN was atrocious: it jumped to 5-6 seconds which I couldn't handle.

Modifié par pope_leo, 20 décembre 2012 - 06:53 .

[Rolo Kipp]

<taking notes...>

So moving the enhanced services management to the Package Manager, rather than the Launcher, if I get it right. Basically, splitting NwNCx in two (Enhanced services and runtime environment - i.e. you're still wanting *some* of the NwNCx functionality to get loaded and running, but want to put engine modification (plug-ins, enhancements, etc) in a different section). I don't have a problem with that. Suits me fine, in fact. =)

Other viewpoints? Vm? Baaleos?

sha1? Link? Interested :-)

<...and running from bards>

[leo_x]

In that scenario enhanced services like NWNCX and custom clients would just be different resources packages that people could install if they wanted to or if a server needed them to. Nothing in NWNCX is necessary to cover today's most basic use cases as far as loading server resources and launching servers, so far as I can tell.

sha1 link. It's what git, etc, use to ensure data in their versioning schemes.

Modifié par pope_leo, 20 décembre 2012 - 07:14 .