50 réponses à ce sujet

[Baaleos]

Btw guys, it looks like the scrape finished successfully.
It claims to have captured all 9000+ topics.
I will do a few tests to be sure, then I will make the mdf (ms sql database) available for download. It's currently at over 500mb in size,
I can also run this against other forums too, it's been suggested to capture the nwn2 forums.
Shall i capture to the same db (increasing its size) or to a new one?

[Faerinaal]

Hi Baaleos, awesome work man.  I would vote for a separate database for NWN and NWN2 data, respectively. Cheers!

[PLUSH HYENA of DOOM]

Thanks, Baaleos...

It would be a shame to see it all just cease to exist.

 

Vive la intransigence!

[Baaleos]

On checking it looks like it might have missed a few posts.

I am going to re-run it, this time adding a message hash field to the post entity.

This way, I should be able to more efficiently detect the presence of existing posts in the database, and be able to append to topics, instead of having to regenerate them from scratch.

 

It was unlikely to be completed perfectly the first time around.

This time when it runs, it will have to generate the topics and posts again from scratch, but it should be the case that it will allow subsequent re-runs to append, rather than replace.

Meaning subsequent re-runs should be fast.

 

I was a little concerned about the db size - given it didnt seem to get all of the topic posts, it was still half a gb.

Im wondering if that is space taken up by Microsoft transaction info.

[Grani]

I'm definitely FOR migrating to the Vault. We don't want to move to another place we have no control of.

 

We've been there already. We've struggled to rescue content that was about to be deleted from the old Vault exactly because it was in someone else's hands, IGN's.

Now the same story happens with the forums, once again because someone else made a decision to shut everything down.

 

Let's not move to another place where this can happen. NWVault truly belongs to the community, that's what we should call "home".

[Baaleos]

It would be great if someone could take the db I am making, and have it display at a new web site, like it was the bioware forums.

Eg: Being able to browse topics etc

 

The database structure I have implemented should make that possible.

 

It might be possible to write a script to migrate the ms sql db I am making to mysql and then have it displayed by an SMF forum.

Some pre-requisite steps would be needed.

Eg: Create dummy users for all the posters.

These get mapped to the posts (author)

Then its a simple case of creating topics, and then creating posts in those topics with the relevant data - which we already have.

[Asymmetric]

Damn, that sucks. I think I'll need to take some the stuff I wrote, put it into some sort of custom content guide and upload it to the vault. At the least for inventory icons.

[henesua]

The forums on the neverwintervault are active.

[Killmonger]

:angry:Grrrr !

 

 

Bloody technocrats and their notions of new and improved....

 

It cost them very little and garners great loyalty to their brand over many years.

A centralized place that everyone (of many generations) can relate to, from numerous platforms.

No, instead, because of "progress" lets scatter the info to the four corners of the net and rely upon other "new" services to gather the data.

Stupid. 

 

 

After all these years,

I am irritated beyond description that Bioware would close these legacy forums.

 

Whatever can be salvaged from these forums needs to be protected and enshrined for future players.

 

A tremendous "Thank you" to all of you who have the technical skills to make it so.

 

Fantastic amounts of love and care have already gone into our beloved game(s).

It would be terrible to waste the enormous trove of technical know how that has been freely contributed by the community.

 

For myself, I am devoted to the Nwn experience.

There has not been another game like it since it was published.

It is not surprising that the powers that be would like it obscured.

Let us confound them.

Long live Nwn

 

I implore the community at large to widely communicate solutions to this problem and enact them quickly.

I shall always support my game community any way that I can.

But let us work together in a co-ordinated manner.

[Baaleos]

Just another update.

I left it running to about 60% completion, but then saw that it was consuming like 600mb of memory.

Microsoft tells you 'oh you don't have to worry about memory management, the .Net framework does it for you.....'

But they lie!!

 

Working with .Net List<> objects- iterating through them in a foreach() statement prevents you from disposing of them - freeing memory.

So I changed it to be more array like.

Eg: Instead of foreach() I am now doing for(i=0;i<sub.topics.count;i++){ Topic t = sub.topics[i]; }

 

The benefit of this is that I can actually dispose of the 'Topic' classes when I am finished committing them to a database.

Now the memory management seems to be a lot cleaner. It goes up and down from 40mb to 80mb, but always returns to around 40-50.

In addition, I have increased the parallel thread count to 25, so it can work on up to 25 topics simultaneously.

So it should be able to reacquire where it was before I turned it off earlier soon.

It was about 60% earlier when I turned it off (after 5 hours of working), I just turned it back on about 10 minutes ago and its about 15% complete.

Further more - it will download offline copies of the Posts to the hdd - so it doesn't have to constantly go back to the bioware forums to grab the information.

This is particularly useful for the top and resume functionality.

 

There has been a lot of messages asking for it to be run against Neverwinter Nights 2 forums.

I will see if I can get it setup to work on those forums too.

[Baaleos]

 

Im running the Nwn 1 Archive on my server, while doing the Nwn2 archive on my desktop.

Its going fast enough, just started the Nwn2 archive and its already about 5-8% done? (guessing based on the % bar)

It has less topics, but possibly more posts.

 

The data for Nwn2 is being put into a separate DB from the Nwn1 - so we can work with them separately.

[Proleric]

OldTimeRadio confirms that if we can scrape content in html format, he will be able to load it into the Omnibus when time permits.

 

This doesn't preclude other archives, too, of course.

[Neverwinter_Knight77]

I don't remember whether I ever retrieved my NWN CD keys. I think I did, but in case anyone forgets, now's the time (just don't ask me how, cuz I don't remember).

[Empyre65]

I don't remember whether I ever retrieved my NWN CD keys. I think I did, but in case anyone forgets, now's the time (just don't ask me how, cuz I don't remember).

That became impossible years ago when the old forums went down.

[Baaleos]

The scraper is starting to annoy me now.

Both the NWN1 and NWN2 scrape jobs finished, for some reason, it only captured 50% of the topics.

It is weird, it is like exactly 50% of the topics, not 57% or 34%, its like 50% on the dot.

There must be some sort of logic issue in my code.

 

As for the Omnibus - im capturing the html content of each post - so it should be possible to string it all together to construct a topic page.

Eg: I am not capturing the html content of a full page, I am literally capturing the html content between the start of one post and the begining of the next post.

[Baaleos]

Im such an idiot - I was doing a for(i=0;i<= ;i++)  loop, and inside the loop, I was still doing i++  - which means that every other topic was being missed.

Duh...

Should have everything scraped soon.

[henesua]

wget is your friend, right? don't ignore your friend.

[Baaleos]

Im just using WebClient.DownloadString(); Lol

[azaz1234]

I suggest the new vault as the best place for future discussion.

 

It's controlled by the NWN community. Most of us are there already.

 

The Nexus isn't a friendly place, in my experience. I don't think a relatively open community like ours would feel welcome there.

 

The forums on the neverwintervault are active.

 

yes, my mistake. for some reason i think of the new vault as playing as similar role as its predecessor -- hosting content but not well suited to forums. but you're right, it could be the ideal place, and the forums are active.

[Cassandra Saturn]

guys, you can go to http://biowaresocial...ork.boards.net/ our new home

[Baaleos]

The NWN 1 and 2 Forum scraping is now complete. (Took a few attempts)

Nwn 1 forum scrapes in at 400mb in size

Nwn 2 forum scrapes in at 282mb in size

 

I am not going to be hosting this for everyone to download, as I get charged for Download bandwidth.

So a million people downloading this from me will consume GB's of data transfer.

 

Can someone 'popular, well liked and trustworthy' contact me with a view to getting a download link off me?

[werelynx]

Might I suggest uploading what you have as a project to new vault? From there any and everyone interested can download the raw data.

[Baaleos]

Suppose that could work - I will see if I can get that done shortly.

 

http://neverwinterva...her/save-forums

 

I've uploaded the 7zip compressed forum scrapes.

7Zip nicely compresses them both to around 17-25 mb.

[Proleric]

I sent the link to OldTimeRadio for addition to the Omnibus when time permits.

[Baaleos]

Im going to do a bit of work to see if I can build an integration to transfer all these posts into an SMF forum,

Best case scenario - it gives us an interactive environment to continue conversations and threads.

Worst case scenario - it gives me a forum that looks popular with 100,000+ posts. Hehe

 

PS - I am running scraper against Dragon Age Inquisition Forums now.

Already spotted a niche part of those forums that broke the scraper - the tech support forum, did not go to a real forum, it goes to the support pages.

I have patches the tool to accommodate.

Also the Baldurs Gate forums behave differently too.

They don't do

 

Forum -> SubForum -> Post -> Topic

Its like the Baldurs Gate forums are the Sub Forums - so they have no main forum.

Which breaks the logic I built into the app.

 

I might be able to code it to test against the forums to determine which mode to run.