Hmmm

[nullstar]

Easy way to find out - copy it to your linux box, unzip if required, and use 'tail' to view the last few lines. If it's truncated you will see some lines at the bottom about PHP execution time being reached.

OK -- I'd forgotten I'd previously installed sshd, so fired that up and the file's copying but looks like it won't complete until a few minutes after I leave. GMT adjusted, I get home from work tomorrow, so expect an answer in the morning.

I'm happy to upload either way (unfortunately have nowhere to host a file that large myself, even temporarily), in case you want the latest snapshot even if the recovery has to be manual. How recent was Malc's backup?

[dbd]

Already tried the archiver then.........

Don't think it works with that timeout type of error anyway, I had thought it was just that what was online is now offline, which it is usually good at finding.

[Akira]

I have to check, I MIGHT have an OLD backup of the forum but don't expect much.

Gamespy have DELTED all the stuff in our old dynamic location, they have destroyed it, it's their fault. I will do MY BEST to not let this go without some sort of...errr... vengeance They really have to pay for it, they destroyed years worth of mesages, they have no right to do so.

Current forum looking good, yeah. evn if phpBB At least if thre's people responsible behind it, if phpBB fucks up, we have somewhere to complain.

And bring on the donation.

[Thunder Force]

Gamespy have DELTED all the stuff in our old dynamic location, they have destroyed it, it's their fault. I will do MY BEST to not let this go without some sort of...errr... vengeance They really have to pay for it, they destroyed years worth of mesages, they have no right to do so.

Why on earth did they delete our data? Or are they blaming it on a virus/worm?

[nullstar]

My Dec. 24 backup is incomplete, too. Bloodflowers: if you're wanting to attempt a recovery of whatever data might be there, I'm more than willing to deliver a copy to you if you'd like the latest we've got...and if you have somewhere to dump it. I'm working nearly 60 hours a week as it is, so I may take a whack at trying to find something to salvage out of this, but it's on no timetable and it may not make sense to duplicate efforts unless you're simply gung-ho about "instantly" restoring whatever you can. LMK what you're attempting, though, so I don't needlessly try to focus on something you are already working on -- assuming I get any time at all.

[system11]

My Dec. 24 backup is incomplete, too. Bloodflowers: if you're wanting to attempt a recovery of whatever data might be there, I'm more than willing to deliver a copy to you if you'd like the latest we've got...and if you have somewhere to dump it. I'm working nearly 60 hours a week as it is, so I may take a whack at trying to find something to salvage out of this, but it's on no timetable and it may not make sense to duplicate efforts unless you're simply gung-ho about "instantly" restoring whatever you can. LMK what you're attempting, though, so I don't needlessly try to focus on something you are already working on -- assuming I get any time at all.

Well - I did a few test runs restoring the database into a mysql one, after editing the dumpfile so that it would load in without bombing out. You end up with lots of posts attributed to userID by number and topic and so on. Obviously, this is a huge can of worms as we can never hope to work out what all the old userIDs were. My suggestion was to basically write a script that pulls the raw text of the posts on a thread out, and dumps them into textfiles. Someone could then pick the worthwhile ones out, and post them in webspace somewhere, so the main part of the score threads (for example) is available to be pasted into a newly created topic here.

The idea of merging the old posts in, is IMHO a road to great folly. As much as anything, the forum is now a moving target as people post (see how busy it got already...). I don't have much time either, but I'm more than willing (and able) to create the above mentioned html files. I might even be able to sift them out linking forum ID and posts, so we can split out the score ones and strategy seperately from the gossip.

[Thunder Force]

Well - I did a few test runs restoring the database into a mysql one, after editing the dumpfile so that it would load in without bombing out. You end up with lots of posts attributed to userID by number and topic and so on. Obviously, this is a huge can of worms as we can never hope to work out what all the old userIDs were. My suggestion was to basically write a script that pulls the raw text of the posts on a thread out, and dumps them into textfiles. Someone could then pick the worthwhile ones out, and post them in webspace somewhere, so the main part of the score threads (for example) is available to be pasted into a newly created topic here.

If we have most of the old forum data in textfile format like that, I'm prepared to attempt sorting through the textfiles and doing the lengthy editing process to save the good content into some readable form. I might even be able to recognize/deduce many of the original authors of the material even if the posts are attributed to userID by number. It is important not to throw away any of the usable raw data we have now.

(Based on the manual topic indexing I was doing before the forum started glitching out, I saw there were just as much good content hidden within various Shmups Chat threads as in the better-signal-to-noise-ratio boards Strategy / Hi-Score).

[Akira]

Listen up
I got access again to all the old files, Gamespy have restored them all for me. I can lift them off the FTP instead of the stupid phpBB backup funciton. Could this be of any use? I think the Strategy, hardware and Scores sections could use a backup with the old data.

BloodF get back to me through PM if this is of any use to you.

[Thunder Force]

Listen up
I got access again to all the old files, Gamespy have restored them all for me. I can lift them off the FTP instead of the stupid phpBB backup funciton. Could this be of any use?

From the sound of it that might be a fullly usable backup of our old boards, without the corruption issue introduced by the php backup-function's data transfer timing out. If you can FTP all of that without your FTP session timing out then that data could be the magic bullet we've been looking for.

Let me know if you need help FTPing the data, I have a decently fast connnection and access to a DVD-writer that could be used to make a read-only archive disc of the old data in case we accidentally corrupt it trying to restore it or something.

If that data turns out usable then perhaps import it as a separate database instance and make it a read-only archive of the old site hosted alongside this one, that way we can still link to and refer to it for its content but it would be a static archive. (This might get around the problem of duplicate userID numbers between the two databases belonging to different users.) Or an export to HTML if it can be cleanly done, which would have the advantage of being easily Google-indexable for searches.

[oceanfr]

cool, hopefully ill be able to check my old PM's

[Trevor spencer]

man this sucks , i posted my highest score on guwange just before the forums went down 620-- something

now it looks like all that hard works gone

[Akira]

Just post it again ;')

It seems this is more hassle than what's needed, having iin mind the forums are fully active again. However we will try to salvage certain parts f the forum.

I'll post here when I do this.

[uwfan]

could you salvage users post counts

[Akira]

No, and I don'tt think that's a valuable thing to salvage.

I'm more interested in the data of hardware and some other stuff, and the PMs.

[Thunder Force]

It seems this is more hassle than what's needed, having iin mind the forums are fully active again. However we will try to salvage certain parts f the forum.

The forums may be fully active again, but it's a far cry from the thousands of pages of shmup-related information our community once possessed at our fingertips.

I appreciate it's a hassle Akira, but if we have a full backup from Gamespy when we should do something with it (e.g. dump to HTML, etc, as above). We can probably drop the old Off Topic content. Yeah, even with some of us prepared to help, this salvage operations could take weeks. But it literally took hundreds of people years to build up that data...

could you salvage users post counts

No, and I don'tt think that's a valuable thing to salvage.

Agreed

[Zach Keene]

Or an export to HTML if it can be cleanly done, which would have the advantage of being easily Google-indexable for searches.

Fairly simple, in theory. If we have a full database backup, set it up on a server somewhere (a local machine if possible) and unleash wget on it. Something like:

Code: Select all

wget -mkpE http://wherever.whatever/forum/index.php

I tested it briefly on the new boards (but didn't let it finish so as not to eat up too much bandwidth...), and it worked pretty well. One problem though is that wget will try to follow every "quote" or "pm" link it finds, so you get a lot of duplicates of the "log in to post" page. (They're easy to delete though.)

Links from the topic indexes didn't work either, but then that may be because I stopped it before the -k option (the one that goes back and fixes links to work locally) could work.

[Akira]

I appreciate it's a hassle Akira, but if we have a full backup from Gamespy when we should do something with it (e.g. dump to HTML, etc, as above).

Hmm if people are capable and willing to do it I will pull it. As it is I can gain access to the FTP but this does not mean I have access to the MySQL database, and I don't have a shell account (I think) to makea dump of it. Any suggestions apreciated.

I thought about re-activating the board and close it (ie no more posts allowed) o people can fetch teh old data and repost it here, and also for people to login, save their PMs to disk and be done with it.

[Zhon]

If the old forum is revived online, or if its files were obtained, I could probably scrape them into something readable. I'm also in favour of the idea of arranging it into a read-only archive. (Maybe we can even extract the best threads while we're at it).

[Zach Keene]

I thought about re-activating the board and close it (ie no more posts allowed) o people can fetch teh old data and repost it here, and also for people to login, save their PMs to disk and be done with it.

That sounds good to me.

[Akira]

I thought about re-activating the board and close it (ie no more posts allowed) o people can fetch teh old data and repost it here, and also for people to login, save their PMs to disk and be done with it.

That sounds good to me.

But with that, how would you insert it back into the board? by hand? What if people start posting duplicate stuff just because the task is not organized and centralized?

Not saying it's impossible, I'm thinking about all the possibilities so we pull it through Im in the preocess of reactivating the account.

[Zach Keene]

But with that, how would you insert it back into the board? by hand? What if people start posting duplicate stuff just because the task is not organized and centralized?

Not saying it's impossible, I'm thinking about all the possibilities so we pull it through Im in the preocess of reactivating the account.

You've got a point. All I wanted was my old high score tables and my Salamander ST, which I was just going to cut-n-paste, so perhaps I'm underestimating how much stuff is would be reposted...

But on the other hand it just seems easier than making HTML dumps or whatever else...

[Thunder Force]

I'm glad we're seeing progress and active discussion on this.

OK, so re-hosting the old reactivated stuff on Dynamic Gamespy may be OK for the short term. In the long run though that's probably a death sentence for the data, knowing how crap Gamespy have been.

So in future that brings us to the option of just FTPing all the restored files from Dynamic Gamespy to the new hosting environment we are running on now. That way it can be rapidly accessed just like the new forum. Probably the host capacity should be plenty for that. I'm not sure what issues hosting two environments side-by-side would introduce, but in theory it sounds feasible.

Then the old stuff could be re-activated in Read-Only mode as a separate http://forum.shmups.com/[b]archive[/b] URL or something similar.

[Icarus]

I'd be willing to help RE: salvaging all the old posts from a knackered backup, attempting to get a clean backup from the dynamic server and/or restarting the forum with all the old info added. It'll be a shame to see all that information disappear in a blink, just because of some crappy dynamic server going to pot.

I do have a backup of my old ST guides in handy BBCode'ed textdoc format (which is how I wrote both my guides), so I'll pop them back up at some point. Unfortunately, that's all I have of the old stuff

Please let me know if I can be of some assistance in the salvage op.

NOTE: I'll be in Venice until Saturday, so I won't be able to reply until then. Sowwie

[llaoyllakcuf]

If anyone could re-post “Where to buy PCBs” in the Trading Station,
that’d be great.

[Akira]

OK, so re-hosting the old reactivated stuff on Dynamic Gamespy may be OK for the short term. In the long run though that's probably a death sentence for the data, knowing how crap Gamespy have been.

´Got me wrong there matey!

As we have seen Gamespy is unreliable as hell (and having in mind i always hated IGN, I'm not really happy of having any of our data hosted with them, but in the case of the site, not the forum, there's not much we can do for now). My idea was to reactivate the forum, locked (ie no new posting would be possible) and make an "all out strike" on it to recover teh data and REPOST IT here in new threads. This would also have teh advantage that people with unsaved, lost PMs can log in and save their PMs to their drives.

I know it's all handmade and primitive but this is teh only solution I can think of. As soon as all salvageable data has been reposted, the forum goes down adn we forget about Gamespy (forum wise, there's still the problem of teh Shmups main site)

[Akira]

Ok im going through al the annoying reactivation/upgrade process now. They woldn't reactivate the account until I upgrade phpBB, because they say it's flawed. They fuck up and they blame US for itheir incompetence!

[Thunder Force]

It's wonderful news that the old forum is intact and can be brought online. But rather than manually reposting threads to here from the forum at Gamespy (which will result in a partial collection of badly formatted and barely readable cut&paste attempts flooding the new forum), wouldn't it be easier to FTP the db over to this host and just bring it online as a separate read-only db instance...

Literally many 10,000s of posts to copy - There might be a 100-500x factor of work difference between the two solutions, in favor of the FTP method.

[Akira]

OK, one of the Gamepsy guys (the only one who has helped throughout all of thismess, thanks man), has gone and made a backup of the DB and put it in ou ftp. This weighs about 33 megabytes RAR'ed, expanding onto 300+ MB.

Now, who do I send this to? And how will this be used with this new forum> ? would it compromise the forum's stability or anytihng else?

[landshark]

If it contains user/pass info, then make sure you send it to someone you really trust.

[Akira]

That data is encrypted anyway, but obviously.