why not link the stuff to your website on a "hidden" link? Something that only you and/or some admin folks can get to?
The Vault Preservation Project
Débuté par
Rolo Kipp
, sept. 28 2012 02:55
Retour en haut
#102
Posté 22 octobre 2012 - 04:36
Rolo Kipp
-
- Members
- 2 791 messages
<staring in dismay...>
It's not the links that are the problem. It's finding a reliable place to store the 500GB+ files somewhere other than IGN. :-/
I could, and may (at least for now) continue with the site re-vamp using direct links to IGN for now, to be fixed later when I find a reliable, always online place to keep the files.
<...at the 10lbs of stuff he needs to fit in a 1lb bag>
It's not the links that are the problem. It's finding a reliable place to store the 500GB+ files somewhere other than IGN. :-/
I could, and may (at least for now) continue with the site re-vamp using direct links to IGN for now, to be fixed later when I find a reliable, always online place to keep the files.
<...at the 10lbs of stuff he needs to fit in a 1lb bag>
#103
Posté 22 octobre 2012 - 05:20
OldTimeRadio
-
- Members
- 1 400 messages
Well, that is a blow but this is just (as Heneusa said) "the first round". I don't have any good suggestions about how to store quite that much information online, save something like a NWN-specific torrent tracker. Or get a 3TB internal/external drive, get the data on there and work on cleaning it up until some kind of better solution is found. Of course, regardless of their merits, neither of those options might be possible in a 48-hour window.
Most of the "solutions" I've focused on were really dire situations where these forums are gone, the Vault's gone, etc. Ideas like shooting CC into Usenet.
But for something a little more realistic, I wonder about the viability of Google Data hosting. It really depends on just how much data is being talked about. Reminds me of working with CC: So many little gears that all need to mesh properly to get something to come out right.
Most of the "solutions" I've focused on were really dire situations where these forums are gone, the Vault's gone, etc. Ideas like shooting CC into Usenet.
But for something a little more realistic, I wonder about the viability of Google Data hosting. It really depends on just how much data is being talked about. Reminds me of working with CC: So many little gears that all need to mesh properly to get something to come out right.
#104
Posté 22 octobre 2012 - 05:48
Rolo Kipp
-
- Members
- 2 791 messages
<meshes...>
Hmmm, Google or perhaps Sourceforge? I'll look into those.
Grabbing the stuff with ACD's scripts works fine, and I will continue to whittle away at them here, over my little Starbucks connection (I have a terabyte external just for the Vault stuff). It's making those files available online that is stumping me.
<...and gnashes his teeth>
Hmmm, Google or perhaps Sourceforge? I'll look into those.
Grabbing the stuff with ACD's scripts works fine, and I will continue to whittle away at them here, over my little Starbucks connection (I have a terabyte external just for the Vault stuff). It's making those files available online that is stumping me.
<...and gnashes his teeth>
#105
Posté 22 octobre 2012 - 05:50
Bannor Bloodfist
-
- Members
- 929 messages
What I mean Rolo, was placing the files/links, all of it, AS part of your website, only with limited access control, IE, private group of users/admins whatever, so that your Host can't complain about the files being stored there.
@OTR, I think the issue now, at this stage, is finding a way to download then re-upload 500GB of files. Most of us have limited (relative term I know) transfer speeds. It has taken Days for Rolo to have his server grabbing stuff without it going down the road to his local PC. That down to local instance and back up is where the main issue is, that and having only 48 hours to "solve" the space usage issue.
@Rolo... I have had the same sort of issue with other webhosting sites, sometimes if you can get a human on the phone, you can resolve it, usually though, they all claim "unlimited" but all actually mean "unlimited in your imagination, once you use it, we delete it!"
@OTR, I think the issue now, at this stage, is finding a way to download then re-upload 500GB of files. Most of us have limited (relative term I know) transfer speeds. It has taken Days for Rolo to have his server grabbing stuff without it going down the road to his local PC. That down to local instance and back up is where the main issue is, that and having only 48 hours to "solve" the space usage issue.
@Rolo... I have had the same sort of issue with other webhosting sites, sometimes if you can get a human on the phone, you can resolve it, usually though, they all claim "unlimited" but all actually mean "unlimited in your imagination, once you use it, we delete it!"
Modifié par Bannor Bloodfist, 22 octobre 2012 - 06:26 .
#106
Posté 26 octobre 2012 - 07:24
Eric of Atrophy
-
- Members
- 4 messages
Reading all this, one has to ask ... the harvesting scripts are working as they should, yes? If so, are we still wanted to harvest things on our own? I have been grabbing willy-nilly as I can, but was just wondering if individual efforts are still warranted, or would be mere redundancy.
#107
Posté 26 octobre 2012 - 07:27
Eric of Atrophy
-
- Members
- 4 messages
By the by, don't know if I've said this before, but I think it is absolutely fantabulous that folks are willing to pull together to preserve content from a game more than 10 years old! Makes me all misty-eyed and reminds me of why I like to hang around NwN in the first place! Sooo ... thank you everyone for the directed efforts to keep access to all that CC, and of course to Rolo Kipp for pressing forward with the Vault duties! Huzzah, y'all.
#108
Posté 27 octobre 2012 - 04:10
Rolo Kipp
-
- Members
- 2 791 messages
<cross...>
At this point, the harvester is working pretty good and we *may* have found a decent host through Michael Darkangel's efforts... still testing it for now.
So, thank *you* for stepping up. That really meant a lot to me personally to see so many people of good heart. However, put your own harvesting on a backburner, if you like. It is redundant for now.
On the other hand, I suspect I will fill up my terabyte external before I finish just the NwN stuff.
<...ing his fingers>
At this point, the harvester is working pretty good and we *may* have found a decent host through Michael Darkangel's efforts... still testing it for now.
So, thank *you* for stepping up. That really meant a lot to me personally to see so many people of good heart. However, put your own harvesting on a backburner, if you like. It is redundant for now.
On the other hand, I suspect I will fill up my terabyte external before I finish just the NwN stuff.
<...ing his fingers>
#109
Posté 27 octobre 2012 - 05:28
Bannor Bloodfist
-
- Members
- 929 messages
I think this was mentioned elsewhere in this thread, but what about the Nexus Mods for Neverwinter as a hosting site? You might/likely would, have to discuss it with them. But I am fairly sure they would be open to the idea of having the files posted there, after all, it would mean much more traffic for them in general.
Of course, you would have to have the Vault's permission to re-post the files elsewhere I guess...
Of course, you would have to have the Vault's permission to re-post the files elsewhere I guess...
#110
Posté 27 octobre 2012 - 05:40
Rolo Kipp
-
- Members
- 2 791 messages
<admiring the pretty view...>
Currently, we're operating with IGN approval and within the IGN terms as a volunteer, community-driven effort to preserve and upgrade the Vault.
I think trying to host the files on an external site that *some* people may think competes (for the record (as I've said before), I firmly endorse redundancy!) with the Vault may cause the whole effort to get scrubbed from the IGN end. I *like* working *with* approval ;-)
There's also the small problem (and I mean small) of getting the Nexus Mods site to allow us to run our harvesting scripts and giving us direct FTP access to the resulting directories. Not a big deal, but a trust issue we don't have if MDA's connection works out.
<...seen from the very edge of the cliff>
Currently, we're operating with IGN approval and within the IGN terms as a volunteer, community-driven effort to preserve and upgrade the Vault.
I think trying to host the files on an external site that *some* people may think competes (for the record (as I've said before), I firmly endorse redundancy!) with the Vault may cause the whole effort to get scrubbed from the IGN end. I *like* working *with* approval ;-)
There's also the small problem (and I mean small) of getting the Nexus Mods site to allow us to run our harvesting scripts and giving us direct FTP access to the resulting directories. Not a big deal, but a trust issue we don't have if MDA's connection works out.
<...seen from the very edge of the cliff>
#111
Posté 27 octobre 2012 - 05:49
OldTimeRadio
-
- Members
- 1 400 messages
FWIW, Dark0ne contacted Maximus directly about this around 6-7 months ago and Maximus indicated that he was still intent on keeping the Vault going. You can find more in the thread in the NWN2 forums about the creation of the Neverwinter Nexus, I don't have the link ATM.Bannor Bloodfist wrote...
I think this was mentioned elsewhere in this thread, but what about the Nexus Mods for Neverwinter as a hosting site? You might/likely would, have to discuss it with them. But I am fairly sure they would be open to the idea of having the files posted there, after all, it would mean much more traffic for them in general.
Of course, you would have to have the Vault's permission to re-post the files elsewhere I guess...
#112
Posté 27 octobre 2012 - 11:10
acomputerdood
-
- Members
- 219 messages
Rolo Kipp wrote...
At this point, the harvester is working pretty good
*ahem*
PRETTY good?
*grumbles*
#113
Posté 28 octobre 2012 - 12:16
henesua
-
- Members
- 3 864 messages
Heh...
The harvester is doing Good by operating well.
The harvester is doing Good by operating well.
#114
Posté 28 octobre 2012 - 02:48
Rolo Kipp
-
- Members
- 2 791 messages
<beating a dead...>
When I get done with the other categories, I'll try it with a set $last.
Other than that, it's been Pretty Damned Good ;-)
<...module>
Well... for some reason the "Get Last" function is returning zilch for modules. *Only* for modules. Very strange.acomputerdood wrote...
*ahem*
PRETTY good?
*grumbles*
When I get done with the other categories, I'll try it with a set $last.
Other than that, it's been Pretty Damned Good ;-)
<...module>
#115
Guest_orizvi_*
Posté 06 mars 2013 - 01:55
Guest_orizvi_*
Guest_orizvi_*
-
- Guests
Has this project died with the hosting issues?
I might be able to assist with the hosting.
I might be able to assist with the hosting.
Modifié par orizvi, 06 mars 2013 - 02:04 .
#116
Posté 06 mars 2013 - 08:54
painofdungeoneternal
-
- Members
- 1 799 messages
Not dead, undead. You can't stop the community, even after the IGN, Bioware, Atari, EA, Obsidian all give up on us as dead, we'll still be coming up with new ways to keep ourselves moving.
I am combining forces, scraping the vault for my own apps purposes, and actually am going to use his server for Never Launcher. Different original purposes, but so much synergy, in that my scraper creates a manifest which can be later used to download, install and merge the original custom content, and I am doing an API to list all the available manifests - which is aimed at being open so others can develop tools using it as well, and i can also setup a drupal integration with that data.
Part of my goals are syncing for authors, so they can easily upload to the vault or nexus just by hitting a button, and likewise downloading for end users ( kind of have to do both to get both using it ). If users want to back up portions of the vault to their hard drive, and the vault goes down, we should be able to then re-upload that content to where it's available and change the manifests so it's still available ( or do a peer to peer sharing system, or perhaps various drop boxs and similar ). I actually hope this never happens though. My preference is for authors to use my tool to download their content from the vault, then transfer it to the nexus, or for new content to upload to both at the same time ( perhaps even adding new places it puts stuff as well ). Regardless, i always hated uploading things to the vault, and intend to make the process hassle free for both authors and end users ( I am lazy enough to do a lot of work to make myself do less work )
And the nexus module manager, which is a mirror of what i am doing, it does not support nwn, nor does it support anything but windows, this should allow such features to work with both the vault, the nexus, and also self hosted content. If they nexus does actually turn on this feature, i should be able to integrate my app with it ( as it's an also open API ).
Lot of pieces to all this, which I have to do one at a time, but the vault is not down yet, and quite a bit is already downloaded via perl scripts.
Might be able to use some bandwidth though for the mirroring some content, as there is no real unlimited bandwidth out there, and we need redundancy. Probably need to discuss more with Rolo about that, big thing is to just get a copy on some hard drive, which we can get a copy of later.
I am combining forces, scraping the vault for my own apps purposes, and actually am going to use his server for Never Launcher. Different original purposes, but so much synergy, in that my scraper creates a manifest which can be later used to download, install and merge the original custom content, and I am doing an API to list all the available manifests - which is aimed at being open so others can develop tools using it as well, and i can also setup a drupal integration with that data.
Part of my goals are syncing for authors, so they can easily upload to the vault or nexus just by hitting a button, and likewise downloading for end users ( kind of have to do both to get both using it ). If users want to back up portions of the vault to their hard drive, and the vault goes down, we should be able to then re-upload that content to where it's available and change the manifests so it's still available ( or do a peer to peer sharing system, or perhaps various drop boxs and similar ). I actually hope this never happens though. My preference is for authors to use my tool to download their content from the vault, then transfer it to the nexus, or for new content to upload to both at the same time ( perhaps even adding new places it puts stuff as well ). Regardless, i always hated uploading things to the vault, and intend to make the process hassle free for both authors and end users ( I am lazy enough to do a lot of work to make myself do less work )
And the nexus module manager, which is a mirror of what i am doing, it does not support nwn, nor does it support anything but windows, this should allow such features to work with both the vault, the nexus, and also self hosted content. If they nexus does actually turn on this feature, i should be able to integrate my app with it ( as it's an also open API ).
Lot of pieces to all this, which I have to do one at a time, but the vault is not down yet, and quite a bit is already downloaded via perl scripts.
Might be able to use some bandwidth though for the mirroring some content, as there is no real unlimited bandwidth out there, and we need redundancy. Probably need to discuss more with Rolo about that, big thing is to just get a copy on some hard drive, which we can get a copy of later.
#117
Posté 06 mars 2013 - 02:32
Rolo Kipp
-
- Members
- 2 791 messages
<stirring his...>
As Pain said, rather like a certain dracolich, we have a phylactery now (about 700gb) and will recover from any rampaging band of adventure... er, difficulties.
We actually have two hosts at the moment, with a third being resurrected and my personal Baby Vault external.
That being said, we would love to have you join us!
There is so much to do before the VPP is a usable mirror and we are all quite busy with the RL that makes our NwN obsession possible ;-P
I urge you to register on the neverwintervault.org and bring your ideas and enthusiasm.
Although it looks like not much is happening, that's my fault, as I haven't posted any updates since last year. I am just getting to the end of an insanely busy period and should have more time to devote to this in a couple-three weeks.
<...old bones>
As Pain said, rather like a certain dracolich, we have a phylactery now (about 700gb) and will recover from any rampaging band of adventure... er, difficulties.
We actually have two hosts at the moment, with a third being resurrected and my personal Baby Vault external.
That being said, we would love to have you join us!
There is so much to do before the VPP is a usable mirror and we are all quite busy with the RL that makes our NwN obsession possible ;-P
I urge you to register on the neverwintervault.org and bring your ideas and enthusiasm.
Although it looks like not much is happening, that's my fault, as I haven't posted any updates since last year. I am just getting to the end of an insanely busy period and should have more time to devote to this in a couple-three weeks.
<...old bones>
#118
Posté 09 mars 2013 - 03:09
Rolo Kipp
-
- Members
- 2 791 messages
<shifting his...>
*moved from the CEP/Q thread because I'd rather not derail that :-P
But how about a batch viewer that will take a model filename grab the model & textures and produce an index screenshot from a command line (so our indexer can call it to snapshot content as it indexes)? Basically a NwN aware renderer; input a model name (model & textures in temp folder), output a 3/4 iso pic?
While crowdsourcing will eventually provide much higher quality pics with, perhaps, details enlarged and anims captured mid-swing, Having a tool to add default, basic images to the catalog would be a fantastic boon. And the tool would only need to run on the indexing machines, so we wouldn't need to require plugins for browsers or anything restricting like that.
What do you think? More probable? Possible? My black hen?
<...relative ground>
*moved from the CEP/Q thread because I'd rather not derail that :-P
Ok. I can accept that.OldTimeRadio wrote...
See, this is a great idea but...Well, I tried this sort of thing out two different ways in the past and IMO, it's one of those "better on paper than in practice" ideas, IMO. I came at it from both VRML and using Acrobat 3D, FWIW. What happens in both cases is you wind up going through a lot of bandwidth, in the form of streamed data or filesize, just to duplicate content (models, textures) that you're not really getting all that much of a payoff for duplicating.Rolo Kipp wrote...
...and (if I can interest someone who has already demonstrated a java3d model viewer into figuring out a web embedded version... ;-) even a model previewer.
IMO, as pedestrian as it sounds, a good 3/4 isometric screenshot made with something like NWN Explorer Reborn with the appropriate "Initial viewing angle YPR" settings (in options) is going to net you something almost as good as a full 3D model with just a fraction of the bandwidth/processing power. And that work can also be crowdsourced much more easily, to boot.
But how about a batch viewer that will take a model filename grab the model & textures and produce an index screenshot from a command line (so our indexer can call it to snapshot content as it indexes)? Basically a NwN aware renderer; input a model name (model & textures in temp folder), output a 3/4 iso pic?
While crowdsourcing will eventually provide much higher quality pics with, perhaps, details enlarged and anims captured mid-swing, Having a tool to add default, basic images to the catalog would be a fantastic boon. And the tool would only need to run on the indexing machines, so we wouldn't need to require plugins for browsers or anything restricting like that.
What do you think? More probable? Possible? My black hen?
<...relative ground>
Modifié par Rolo Kipp, 09 mars 2013 - 03:10 .
#119
Posté 09 mars 2013 - 03:21
OldTimeRadio
-
- Members
- 1 400 messages
In NWN Explorer Reborn, you can right click on a model in the tree list on the left and select "Export as TGA" and it will load the model with your YPR view angle settings in options, etc., and export a screenshot. Takes a little fiddling but it's not that bad.
In 3DS Max there's probably a way to batch out screenshots with a custom MaxScript and the bonus there is you get full renders if you want. I'm not volunteering to do it but if you look through the various scripts that come with NWMax and VelsTools (i.e. Portrait Maker plugin, IIRC), you could probably cobble something together which did it for ya. IIRC, NWMax does not have any problems with memory leaks when loading models so you can probably do as many as you like in a batch.
Edit: Remember, 3DS Max is more like a 3D-oriented operating system than just a 3D modeling program. There is all kinds of things you can hook into or out of with it, using MaxScript.
In 3DS Max there's probably a way to batch out screenshots with a custom MaxScript and the bonus there is you get full renders if you want. I'm not volunteering to do it but if you look through the various scripts that come with NWMax and VelsTools (i.e. Portrait Maker plugin, IIRC), you could probably cobble something together which did it for ya. IIRC, NWMax does not have any problems with memory leaks when loading models so you can probably do as many as you like in a batch.
Edit: Remember, 3DS Max is more like a 3D-oriented operating system than just a 3D modeling program. There is all kinds of things you can hook into or out of with it, using MaxScript.
Modifié par OldTimeRadio, 09 mars 2013 - 03:23 .
#120
Posté 15 avril 2013 - 11:00
Rolo Kipp
-
- Members
- 2 791 messages
<sneaking in a little work...>
I can now say the VPP is minimally functional.
Lots of theme issues, but they are mostly cosmetic.
Special thanks to acomputerdude and Meaglyn for their scripting expertise, Tarot for keeping at me and Kamal & Sagard for letting me experiment with their projects :-)
<...between hours of..., er, work>
I can now say the VPP is minimally functional.
- Artists can create projects or write articles of various categories.
- The projects and posts can have images or files attached and formatted.
- Both images and files may be uploaded from the artist's machine, transfered in from a URL or FTP'd to the site dropbox and attached - all without requiring intervention from (sometimes absent-minded) moderators.
- The forums work (still have to fix the editor background :-P ).
- The Contact form works.
- The indexing and search works.
- And the chatroom works (barely).
Lots of theme issues, but they are mostly cosmetic.
Special thanks to acomputerdude and Meaglyn for their scripting expertise, Tarot for keeping at me and Kamal & Sagard for letting me experiment with their projects :-)
<...between hours of..., er, work>
#121
Posté 15 avril 2013 - 11:04
Rolo Kipp
-
- Members
- 2 791 messages
<leaving a note...>
<...for his absent friend>
I have Torlacks source and I'll be trying to work something into my Region Editor.OldTimeRadio wrote...
In NWN Explorer Reborn, you can right click on a model[/b] in the tree list on the left and select "Export as TGA" and it will load the model with your YPR view angle settings in options, etc., and export a screenshot. Takes a little fiddling but it's not that bad.
...
<...for his absent friend>
#122
Posté 04 mai 2013 - 03:12
Rolo Kipp
-
- Members
- 2 791 messages
<quite happily...>
Ok, another call for help :-)
I'd like to get all the various patches up on the VPP so they are available ASAP.
The current bottleneck is my paltry Starbucks wireless upload speed. So, if you have a copy of some of the patches and are willing, please upload them to the anonymous FTP (no password) at ftp.neverwintervault.org.
Once they are in the anon dropbox on the VPP, I can attach them to the projeect pages (this is the same mechanism you will use for project files larger than 128mb).
To prevent stepping all over each other, please declare a block of patches here and only here:
I am currently uploading (expected to take all day) the three linux clients (about 1.5gigs total)
Anyone for the NwN1 patch? SoU? HotU?
NwN2 English?
NwN2 French?
etc?
<...quite busy>
Ok, another call for help :-)
I'd like to get all the various patches up on the VPP so they are available ASAP.
The current bottleneck is my paltry Starbucks wireless upload speed. So, if you have a copy of some of the patches and are willing, please upload them to the anonymous FTP (no password) at ftp.neverwintervault.org.
Once they are in the anon dropbox on the VPP, I can attach them to the projeect pages (this is the same mechanism you will use for project files larger than 128mb).
To prevent stepping all over each other, please declare a block of patches here and only here:
I am currently uploading (expected to take all day) the three linux clients (about 1.5gigs total)
Anyone for the NwN1 patch? SoU? HotU?
NwN2 English?
NwN2 French?
etc?
<...quite busy>
Modifié par Rolo Kipp, 04 mai 2013 - 03:17 .
#123
Posté 04 mai 2013 - 07:29
The Amethyst Dragon
-
- Members
- 1 878 messages
I can grab and upload the three NWN Windows patch files today.
Other NWN patch files I know of, for those looking for them.
Other NWN patch files I know of, for those looking for them.
#124
Posté 05 mai 2013 - 02:22
The Amethyst Dragon
-
- Members
- 1 878 messages
Got the Windows ones uploaded to the ftp address. I'm going to go ahead and grab the Mac ones, and the German and Polish unofficial ones, tonight and ftp those as well.
#125
Posté 05 mai 2013 - 04:03
Rolo Kipp
-
- Members
- 2 791 messages
<getting his copper's worth...>
Ok burning up the wireless at HQ. Uploading second linux client now.
You can see what's uploaded already on the Patch Catalog.
Edit: TAD did all the mac clients and I've done the linux clients. Working on the standalone servers (Linux & PC) now.
<...out of his favorite pub>
Ok burning up the wireless at HQ. Uploading second linux client now.
You can see what's uploaded already on the Patch Catalog.
Edit: TAD did all the mac clients and I've done the linux clients. Working on the standalone servers (Linux & PC) now.
<...out of his favorite pub>
Modifié par Rolo Kipp, 05 mai 2013 - 06:48 .
