45 réponses à ce sujet

[kamal_]

"NWN2HakpaksOriginal Range:1-590 Have up to: 482 ( probably done, Kamal is doing )"
Complete.
"NWN2HakpaksCombined Range:1-62 Have up to: 62"
Also done.
"NWN2Models Range:1-240 "
Complete.
NWN2 Tools: 1-104
Complete.
NWN2 Plugins 1-114
Complete.
NWN2Tutorials Range:1-141
Complete.
NWN2Strategies Range:1-11
complete
NWN2ModulesEnglish Range:1-531 Have up to: 529
530-531 complete
NWN2Gameworlds Range:1-275 Have up to: 274
275 complete
NWN2VisualEffects Range:1-73
complete
NWN2Textures Range:1-51
complete
NWN2UI Range:1-174
complete
NWN2Scripts Range:1-409
complete

NWN2PWC Range:1-141
complete
NWN2Characters Range:1-114
complete
NWN2Portraits Range:1-97
complete
NWN2PrefabPlaceables Range:1-167
complete

NWN2Ideas Range:1-487
complete
FanFiction Range:1-571
complete

Hakpaks Range:1-8218 Have up to: 7849 ( Need that last 500 )
completed last 500
Models Range:1-505
complete
Creatures Range:1-691
complete

Artwork Range:1-1039
in progress
Ideas Range:1-3174
in progress
Screenshots Range:1-500
complete
NWN2Articles Range:1-300
complete
NWN2Reviews Range:1-54
complete
NWN2Links Range:1-41
complete
NWN2UserScreenShots Range:1-362
complete

Other Range:1-1544
in progress
Prefabs Range:1-1238
in progress
Movies Range:1-639
in progress

Modifié par kamal_, 28 juin 2013 - 03:10 .

[niv]

in progress:
nwn2prefabareas 1-637

[niv]

@brian: I think the filesize parser for attachments is broken. 28.x kb -> 28, 18.4MB -> 18.

Can be seen here: http://api.neverwint...prefabareas/53. Should probably read 18874368 + something bytes?

[painofdungeoneternal]

Good catch.

I can fix that after the fact manually ( or rather by sql updates ), i have the file and it's bytes on another server focused on just the files. The md5 and sha1 are the main things i need to generate.

[meaglyn]

Textures Range:1-164 completed
nwn1 sounds 1-426. will continue unless someone else gets through them all first.
NWN1 links 1-291 completed
nwn1 scripts 1-3884: starting at 2605

Modifié par meaglyn, 28 juin 2013 - 06:50 .

[Tchos]

Working on the NWN1 range of uncollected scripts, but it's not downloading the files from every page. I have the first 4 boxes ticked, and the last two empty.

For example, script 2651, Djinn Lamp, has a file to download called Genie.zip, which does not appear in the PackageContents folder, although I can download it manually.

Is this a problem?

Addendum: NWN1 Scripts Range:2606-3884 complete, but unknown how many files may not have been saved.

Addendum 2: 
NWN2Other Range:1-297 complete

Modifié par Tchos, 28 juin 2013 - 08:16 .

[niv]

See posting further down for current info.


Alright, so I wrote a little verify script that runs against a category and id range.
Here's the results for a check on nwn2prefabareas, which I scraped last evening.
There weren't any errors in the scraper log output.
Edit:
I verified those 400 some more and there's some that aren't in the API but should.
For example:- http://nwvault.ign.c...as.Detail&id=23vs - http://api.neverwint...2prefabareas/23
Some of the missing files are:* urls to other vault entries* hosted on external servers that are down since
I haven't checked them all though, so I don't know if there are missing files that are not falling into one of those two categories. I don't feel like hitting the vault with my own http parser yet but that may be not a bad idea.
Edit again:
I took the liberty of running that verify script against nwn2other and it found some files that are clearly on the vault but not in the API. Seems that some files aren't uploaded properly.
Example:- http://nwvault.ign.c...r.Detail&id=294 has "Cormick_Dialog_Fix.rar"- http://api.neverwint...t/nwn2other/294--> D9F23B4A31FB801E75DF8EC171CDEAC1C403FD3C- http://vpp.tbotr.com...403FD3C/exists/ doesn't exist* rest of verify output

Modifié par nivviv, 29 juin 2013 - 10:28 .

[Tchos]

NWN2Articles Range:1-300 complete

[kevL]

It's getting harder to find categories that need to be processed.


here's what i been working through:

nwn2portraits 1-97
nwn2characters 1-114
nwn2prefabplaceables 1-167
nwn2 other 1-297 (hang @ 19, 212)

nwn1 portraits 1-6413 (hang @ 3219, 5682)
nwn1 models 1-505
nwn1 creatures 1-691
nwn1 sounds 1-882 (hang @ 427, 554, 592, 711, 732, 878)


What i noticed about the scaper hangs I got, is when there were DLs of 100+ mb files. Or when there were lots of 10+ mb files. Anyway, I believe Pain tracks the missing spaces best; will wait for an update.

Modifié par kevL, 28 juin 2013 - 01:54 .

[kamal_]

Should bring us up to date on what's complete, in process, needs work. Will try to keep this updated so Pain can focus on more important things.

edit: All categories are now complete.

• Modules Range:1-6397 Have up to: 6393 ( need latest modules ) ( completed last 4 Kamal )
• Gameworld Range:1-1392 Have up to: 1392 ( complete )
• Hakpaks Range:1-8218 Have up to: 7849 ( Need that last 500 ) (completed last 500 Kamal)
• Models Range:1-505 ( complete kevL )
• Creatures Range:1-691 ( complete kevL )
• Scripts Range:1-3884 Have up to: 2605 ( complete meaglyn )
• Characters Range:1-1451 Have up to: 50 ( in progress Kamal )
• Portraits Range:1-6413 ( complete kevL )
• Prefabs Range:1-1238( complete Kamal )
• Sounds Range:1-882 ( complete kevL )
• Textures Range:1-164 ( complete meaglyn )
• Ideas Range:1-3174 ( in progress Kamal )
• Screenshots Range:1-500 ( in progress Kamal )
• Movies Range:1-639 ( complete Kamal )
• Links Range:1-291 ( complete meaglyn )
• Community_News Range:1-2999 ( in progress Kamal )
• Other Range:1-1544 ( in progress Kamal )
• NWN2ModulesEnglish Range:1-531 Have up to: 529 ( complete Kamal )
• NWN2ModulesInternational Range:1-75 Have up to: 75 ( complete )
• NWN2Gameworlds Range:1-275 Have up to: 274 ( complete Kamal )
• NWN2HakpaksOriginal Range:1-590 Have up to: 490 ( Complete Kamal )
• NWN2HakpaksCombined Range:1-62 Have up to: 62 (Complete Kamal )
• NWN2HakpaksModuleSpecific Range:1-42 Have up to: 42 ( complete )
• NWN2PWC Range:1-141 ( Complete Kamal )
• NWN2Models Range:1-240 ( complete Kamal )
• NWN2Tools Range:1-104 ( complete Kamal )
• NWN2Plugins Range:1-114 ( complete Kamal )
• NWN2Characters Range:1-114 (complete kevL)
• NWN2Portraits Range:1-97 (complete kevL)
• NWN2PrefabAreas Range:1-637 ( complete nivviv )
• NWN2PrefabPlaceables Range:1-167  (complete kevL)
• NWN2Scripts Range:1-409  ( Complete Kamal )
• NWN2UI Range:1-174 (complete Kamal )
• NWN2VisualEffects Range:1-73 ( complete Kamal )
• NWN2Textures Range:1-51  ( complete Kamal )
• NWN2Tutorials Range:1-141 ( complete Kamal )
• NWN2Strategies Range:1-11 ( complete Kamal )
• NWN2Ideas Range:1-487 ( Complete Kamal )
• NWN2UserScreenShots Range:1-362  ( complete Kamal )
• NWN2Movies Range:1-168 ( complete Kamal )
• NWN2Articles Range:1-300  ( complete Tchos )
• NWN2Reviews Range:1-54  ( complete Kamal )
• NWN2CommunityNews Range:1-350 ( complete Tchos )
• NWN2GameInfo Range:1-122 ( complete Kamal )
• NWN2Links Range:1-41 ( complete Kamal )
• NWN2Other Range:1-297  ( complete Tchos )
• FanFiction Range:1-571  ( complete Kamal )
• Portfolios Range:1-5429 ( in progress Kamal )
• Artwork Range:1-1039 ( complete Kamal )

Modifié par kamal_, 30 juin 2013 - 10:11 .

[Tchos]

NWN2Other is also complete.  [Edit: Thanks for adding to the list]

I'll get on the NWN2 community news next.

Edit: NWN2 Community News complete.

Modifié par Tchos, 28 juin 2013 - 06:40 .

[meaglyn]

Links Range:1-291 completed
Scripts Range:1-3884 Have up to: 2605 : completed to 3884

[kamal_]

One of the things Pain would like to get is the actual html of the Vault. The scraper saves this. In your Neverlauncher folder, there is a folder called VaultContents. When you are done scraping, if you could zip this up and post it somewhere, this way we'll have access to the Vault's actual html pages if necessary in the future.

[Tchos]

Hang on, it's not uploading that when I scrape? The HTML is all there is of some of these sections, like the articles, ideas, reviews, and a lot of scripts.

[kamal_]

Tchos wrote...

Hang on, it's not uploading that when I scrape? The HTML is all there is of some of these sections, like the articles, ideas, reviews, and a lot of scripts.

My understanding is the scraper program puts the download from the page in a file in the storage service, and creates appropriate Vault 2.0 html for it. The contents of the folder are the actual html pages of the Vault, so getting it is more of a "just in case", as well as making any future rescraping go faster. If you want a specific and correct description, hop onto irc and Pain can explain exactly.

[painofdungeoneternal]

It is uploading the data on those pages, getting the sha1's, and it's all going into a very large database. Basically its transforming it's format into something a lot easier for a computer to deal with.

However if there is a bug ( and with 17,0000 + projects there is going to be a bug ) and if the vault is gone, i can adjust my code and rescrape it with those pages. I also am still adjusting things to get better data out of the files. ( for example the last issue was the encoding, projects with accents above their letters were not working, so i had to fix that and redo the scraping and now those foreign characters show up )

If the vault is up for the next year i don't need them, but since you did the work, preserve that, and at your convenience ( no hurry ), just in case the plug is pulled soon, that is our last contact with all that data.

[Tchos]

Okay, will do.

• nwn1 scripts
• nwn2 articles
• nwn2 community news
• nwn2 other
• nwn2 scripts
• nwn2 tutorials

This post looks suspiciously like a link farm.

Modifié par Tchos, 29 juin 2013 - 05:08 .

[niv]

 Here's what I have verified so far. The "failed" category is:

• entries that are 404 on the API but should be there (the vault page exists)
• entries on the api that don't have hashes, but have accessible files on the vault
• entries on the api that HAVE hashes but neither of the two configured file stores knows about

The verify script assumes external links to be ok (even though they might not be).

Here is the detailed breakdown of all test runs. You can extract the date for when each category was verified from the gist revision dates (I'm lazy today).

• Modules:
• Gameworld:
• Hakpaks: 8218 verified: 5268 ok, 2950 failed.
• Models: 505 verified: 486 ok, 19 failed.
• Creatures: 691 verified: 663 ok, 28 failed.
• Scripts:  3884 verified: 3678 ok, 206 failed.
• Characters:
• Portraits: 6413 verified: 6114 ok, 299 failed.
• Prefabs:
• Sounds: 882 verified: 806 ok, 76 failed.
• Textures: 164 verified: 155 ok, 9 failed.
• Ideas:
• Screenshots:
• Movies:
• Links:
• Community_News:
• Other:
• NWN2ModulesEnglish:
• NWN2ModulesInternational:
• NWN2Gameworlds:
• NWN2HakpaksOriginal:
• NWN2HakpaksCombined:
• NWN2HakpaksModuleSpecific:
• NWN2PWC:
• NWN2Models:
• NWN2Tools: 104 verified: 12 ok, 92 failed.
• NWN2Plugins:
• NWN2Characters:
• NWN2Portraits:
• NWN2PrefabAreas: 637 verified: 615 ok, 22 failed.
• NWN2PrefabPlaceables:
• NWN2Scripts:
• NWN2UI:
• NWN2VisualEffects:
• NWN2Textures:
• NWN2Tutorials:
• NWN2Strategies:
• NWN2Ideas:
• NWN2UserScreenShots:
• NWN2Movies:
• NWN2Articles:
• NWN2Reviews:
• NWN2CommunityNews:
• NWN2GameInfo:
• NWN2Links:
• NWN2Other:
• FanFiction:
• Portfolios:
• Artwork:

Modifié par nivviv, 29 juin 2013 - 03:33 .

[Rolo Kipp]

<tapping the...>

Fantastic feedback and it really makes me curios what's happening with NwN2Tools.
Are you by chance logging the project cat/Ids?

<...audio-vibratory enhancing device>

[niv]

The category and post ids are in the detailed output gist I linked, reproduced here: https://gist.github....e32902cf9ca07ee

The three fields for each failed file are: id_in_api,sha1_or_empty,filename

sha1 is empty if the API doesn't have any files stored. It's not empty if the API has a hash stored but the scraper didn't upload any to the file stores I check (vpp.tbotr.com, vss.neverwintervault.org).

400 means the vaultscraper parser probably conked out or something.

All lines in the output are fixable errors (unless there's a bug in my verify script). I'm ignoring external links and vault entries that were rejected ("Not Approved").

Modifié par nivviv, 29 juin 2013 - 01:02 .

[Rolo Kipp]

<plowing the same...>

Starting re-scraping NwN1 hakpaks.
Edit: Moved scraping discussion to Developer forum on the new Vault. 
We now return you to your regularly scheduled Never Launcher updates :-)

<...fields with new tools>

Modifié par Rolo Kipp, 30 juillet 2013 - 04:52 .