36 réponses à ce sujet

[PapaCharlie9]

I started to do a bunch of threads, but then realized that it might be better to wait until after BSN goes read-only and then capture any final edits at that time. Otherwise whatever gets added after the snapshot is taken may not be crawled or archived before the forum shuts down.

Still, a great idea and it's gratifying to see that some of the threads that are most important to me have already been archived by other people!

[colemanshane]

Like others have said before, I'd like to thank you people for your work. You're doing the makers work, 

[Cassandra Saturn]

i finished putting the boards up today. the last was Dragon Age forum.

[Shades of Night]

i finished putting the boards up today. the last was Dragon Age forum.

That's all well and good and you should tell people about it in the topics where people are looking for a list of alternative forums to go to.

 

However in this topic we're largely discussing how to try and preserve the existing Bioware forum posts, which as I mentioned, will be impossible for you to do manually within the limited timeframe that we have. Thus this discussion. If on the other hand you do have any ideas about speeding up the process or alternative solutions for collecting the current forum content then by all means, please share

[Neyjour]

I've started working on:

The BioWare Forum > Dragon Age > Dragon Age Franchise > Fan Creations

 

EDIT: Good lord... I just did the first 2 pages, and the amount of time that took is... *shakes head*  (and there's still 100 pages to go, for that forum alone).  Unless a TON of people are willing to join in and help, saving everything via the Wayback Machine doesn't seem like a viable solution.  Doing it one thread at a time is beyond tedious and incredibly time-consuming. 

 

I know Fextralife (and possibly others) are in the process of attempting to import all (or most) of the threads on these forums with web scrapers.  We may just have to rely on them to save everything before BioWare vaporizes it all...

[Fexelea]

I've started working on:

The BioWare Forum > Dragon Age > Dragon Age Franchise > Fan Creations

 

EDIT: Good lord... I just did the first 2 pages, and the amount of time that took is... *shakes head*  (and there's still 100 pages to go, for that forum alone).  Unless a TON of people are willing to join in and help, saving everything via the Wayback Machine doesn't seem like a viable solution.  Doing it one thread at a time is beyond tedious and incredibly time-consuming. 

 

I know Fextralife (and possibly others) are in the process of attempting to import all (or most) of the threads on these forums with web scrapers.  We may just have to rely on them to save everything before BioWare vaporizes it all...

 

Indeed, manual imports will be very limiting simply because it requires too much time.

 

I am closing in on a definitive answer regarding salvaging posts - I hope it works. Keep your fingers crossed everyone

[Shades of Night]

Yeah, I did it on a muuuch smaller forum but it took months. I was hoping if enough people worked together we might be able to make a dent in this one. But I think it's just too much content to do one page at a time.

 

I have sent an email to the people at the Wayback Machine to see if they'd be willing to grab the whole forums for us, but the response just directed me back to the one page at a time page. I sent another email clarifying so we'll see.

[Neyjour]

Fexelea - If you can get that to work, it'll be an absolute godsend.  Thanks again for everything you've done so far (and are attempting to do).  It's much appreciated.  I'll be keeping my fingers (and toes!) crossed.

 

Shades of Night - If it's possible for them to grab all the forums, or at the very least provide a way for us to do it ourselves in chunks (by entire forum categories, or even just an entire page of threads at a time), that would great.  Even if Fexelea is successful, it would still be good to have a backup at the Wayback Machine. 

[CreeperLava]

HTTrack is making progress on saving forum.bioware.com : it's running since 2 days (had to restart to remove pictures), up to 60 GB of data. I don't think I'm even 10% in yet, but 37 287 pages are saved. I'd say there are more than 400k total. So definitely not doable manually.

[PapaCharlie9]

Indeed, manual imports will be very limiting simply because it requires too much time.
 
I am closing in on a definitive answer regarding salvaging posts - I hope it works. Keep your fingers crossed everyone

 
 

Yeah, I did it on a muuuch smaller forum but it took months. I was hoping if enough people worked together we might be able to make a dent in this one. But I think it's just too much content to do one page at a time.
 
I have sent an email to the people at the Wayback Machine to see if they'd be willing to grab the whole forums for us, but the response just directed me back to the one page at a time page. I sent another email clarifying so we'll see.

It's still worthwhile to archive some hand-picked pages and threads on the Internet Archive, since it has the distinct advantage of preserving the entire look of the original site. It's doubtful that any of the current scraping efforts will look as good.

Not that I'm complaining -- having the content in some form is better than not having it at all. And I'm not suggesting that hand-picked stuff replace the scraping efforts. Having important content stored in multiple different sites is a good thing.

[Tarshana]

I'll start on the Dragon Age Toolset pages. 

Edit: I broke it. After 25 pages it says it's down. Cough. Sorry?   (I'm sure it wasn't me but I find it funny)

[CreeperLava]

 
It's still worthwhile to archive some hand-picked pages and threads on the Internet Archive, since it has the distinct advantage of preserving the entire look of the original site. It's doubtful that any of the current scraping efforts will look as good.

Actually, HTTrack saves everything, not only HTML. It also saves CSS and JavaScript, so the pages are identical to the ones you would see here. It's like the BSN, but offline.