Bad news - downtime / security breach

[system11]

Well, there's no nice way to say this.

The server got rooted, some time between mid December and today (perhaps indeed multiple times, it was being script attacked). I only noticed because today they kicked off a spam script which resulted in thousands of bounces in my mail. They compromised the exim mail daemon to gain root access. This is why the forum was down today for a couple of hours - I simply shut everything down until I could work out what state it was in. The original point of entry (exim) has been fixed.

There is no evidence files (database backups) were taken or tampered with, but at the same time I cannot prove that they didn't do this. Unfortunately this means some hassle for everyone and definitely some downtime.

What needs to happen to the server:
Find replacement
Migrate forum to temporary home
Review forum data for anything suspicious - mostly dodgy links that might have been inserted.
Bring forum back up
Ship old server back to my place, total crash & burn reinstall
Ship old server back to hosting co, replace web files, verifying integrity against pre-December backups
Migrate forum back to server

What this means to users:
A couple of episodes of downtime
Some users may lose their avatars
Passwords cannot be considered secure

What you need to do:
1) If you use your shmups forum password here at other sites, change it at those sites to something different
2) After the initial relocation, change your password here.

This fucking sucks. More news as soon as it happens.

[emphatic]

Damn those hacker scumbags to hell, I say.

[Sumez]

Aren't password always encrypted on PHPBB?
Of course, getting access to the encrypted pass is still a security breach, but brute forcing the password would be a pretty complicated process.

[system11]

Aren't password always encrypted on PHPBB?
Of course, getting access to the encrypted pass is still a security breach, but brute forcing the password would be a pretty complicated process.

Yes, they are encrypted, the danger is brute force as you suggest. It seems they were really just interested in using the machine to spam people, but it seems sensible to change passwords elsewhere. Since the current host cannot be considered trustworthy, it is not worth changing passwords here yet.

[StarCreator]

I'm probably not the best candidate since I only have a shared hosting account, but if you need someone to temporarily host the forums while you work on the server let me know.

[dieKatze88]

I'm probably not the best candidate since I only have a shared hosting account, but if you need someone to temporarily host the forums while you work on the server let me know.

Ditto. I'm on Dreamhost, It's not the greatest, but it does PHPbb pretty decently.

[Krimzon Kitzune]

Well, damn.

Anyways, thanks for letting us know, man. Much appreciated.

[Sumez]

If all you need is a phpBB with a moderate activity level like this, most webhotels should be sufficient - I know I've done much more advanced stuff on mine, what are your requirements for a potential new server?
Of course, it's understandable if you want more control over the server (ie. not being run by an external company)

[system11]

The new server is already installed - I'm just working on apache/exim/php/etc, then it's time to start auditing and migrating the sites. A friend of mine with a very hefty ESX server in the same datacenter as this server kindly created me a virtual machine we can use until I can clean this one down completely and then reformat it.

[Blackbird]

Bummer. Fortunately, I don't think that I use this password for anything important, only other forum-related stuff like this. I'll keep it in mind, though.

[spadgy]

Gonads! And I only changed all my passwords a couple of weeks back!

If I can help at all with any of the legwork System11, let me know mate.

I'm not a techy really, so I'll ask - is it worth backing up things like high score threads (something I do once a month with the ones I maintain anyway) just in case?

[system11]

Right - everything has been migrated to our new (temporary) host.

If anything isn't working, please let me know - it SEEMS to be ok. If small bugs are discovered the server may restart at any time.

All users should reset their passwords now

[Edwards80]

No biggie, but auto login does not seem to be working. You're required to manually log in after returning to the site.

Aside from that, thanks for the heads up and I hope this isn't causing you too much of a headache

[S20-TBL]

No biggie, but auto login does not seem to be working. You're required to manually log in after returning to the site.

Probably disabled for the meantime in case any keyloggers are still wandering around.

[system11]

No biggie, but auto login does not seem to be working. You're required to manually log in after returning to the site.

Aside from that, thanks for the heads up and I hope this isn't causing you too much of a headache

It'll be cookie settings somewhere, I'll look into it - might require force logging everyone out.

[TrevHead (TVR)]

I just posted to say thanks Bloodflowers for putting in your time over the years keeping this place running and for fixing this current problem, also thx to the mods and anybody else providing support

[system11]

No biggie, but auto login does not seem to be working. You're required to manually log in after returning to the site.

Probably disabled for the meantime in case any keyloggers are still wandering around.

It's not on the compromised host anymore - I moved every single service off onto a temporary host kindly provided by a friend.

[mjclark]

Yeah- just to second the appreciation.
Thanks for getting us back on track.
I can't imagine what a massive pain in the arse it must be to sort this stuff out.

[sven666]

I just posted to say thanks Bloodflowers for putting in your time over the years keeping this place running and for fixing this current problem, also thx to the mods and anybody else providing support

I approve of this message.

[moozooh]

bloodf is the man. :)

[njiska]

Ouch, sounds like you've got a ton of work ahead of you Bloodflowers, er system11. Good to know you were able to get everything moved over so quickly.

Fortunately the password I use with shmups forum is different from the one i use for anything that has my credit card tied to it, so I'm not worded about that kind of theft. I think a lot of us probably learned that lesson with the Gawker hack.

Good luck getting it all sorted out.

[Vexorg]

Seems to be working fine here. Fortunately I was just using the auto-generated password on this board, so it doesn't affect anything else. Changed it anyway, just to be sure.

[Aquas]

Thanks for the work getting things back in order. Props.

[Daigohji]

Thanks for the hard work and updates.

[louisg]

Hey bloodf, were the passwords salted?

[system11]

Hey bloodf, were the passwords salted?

You'll have to look up what PHP does by default.

I think I've fixed the session handling problem now, it's a change in PHP which is now the newest version, I've had to put a workaround in for now but the correct fix is going to take a little longer.

[emphatic]

Yeah, thanks a lot for the great work!

[tinotormed]

GRRRRRRR

Those hackers are not only do they become a pain in the a**, but also they can easily impale themselves through the localhost of the sql server of every single website!

[system11]

Someone noted to me that it's probably worth explaining something I took for granted.

I can't prove they didn't take a copy of the database. It seems unlikely given that they wanted to use the box as a spamhost, but it's impossible to prove. Members who regularly use the trading forum might want to check through their saved messages to see if they've sent anything sensitive. I would hope they haven't, it should mostly be ways for other people to give you money, after all.

[drauch]

Good job on getting it back up! I almost didn't know what I was gonna do with myself last night. Shmups forum is my facebook.