EC downtime / lost posts-PMs-blogs

[Chip]

Hi, everyone.

We thought it would be important to explain the problems EC has been experiencing in the last 24 hours.

Starting sometime yesterday afternoon (US central time), the forum started acting erratically. Random database errors were occurring, people were getting messages that they had posted twice in the same thread in a short time, when they had not, attempting to send PMs was causing intermittent database errors, and various other problems cropped up.

The problem was identified about 8pm CT and we started working to resolve the issue. The site was up and down during the night, and the initial fix we did (restoring to the 6pm backup from Tuesday night) did not fix the problem. We subsequently went back to the 6am backup, and that still did not solve the problem. In some cases, posts were appended to the wrong threads, creating even more confusion.

Ultimately, we found there were multiple issues that contributed to the problem, creating a "perfect storm" of problems that contributed to a big mess. It was a combination of issues with the MySQL server and its related caches that manage the database that stores the posts, combined with running a very out-of-date version of vBulletin that likely contributed to the original problems.

So... we have now upgraded to the most current revision of the major version of vBulletin we're currently running (version 3), and restored the database from 6am yesterday. We've also made some tweaks, and (for the time being at least) removed the feature that shows who is online, as, with so many users, that was causing excess load on the server that may have contributed to the problem.

We've tested everything else and believe that we've solved the problems.

What this means to you:

-- Any posts, blog entries, threads, member registrations, wall postings, PMs, or any other activity done between Tuesday 6am and 11pm Central Time tonight (i.e, Wednesday, when I'm writing this) have been lost with restoration of the backup.

-- If you registered between yesterday at 6am and now, you will need to register again.

-- If you see any odd or erratic behavior by the forum software, any error messages, etc., please report them in this thread so we can keep an eye on things and take any action required.

The staff understands how frustrating it can be to have entire messages, threads, blog posts, and so forth just disappear, particularly when you've spent a bunch of time writing them. And the staff lost all of their posts (and the admins, most of today trying to get things fixed) so it's a frustrating situation for everyone. But fortunately, these sorts of problems are pretty rare, and we don't anticipate any further problems.

We appreciate your understanding!

EDIT: Member fiddledeedee has pointed out there's still a way to see who is online, which I've reposted below. Thanks!

You are still able to get a list of people online if you go to Quick Links –> Who's Online (at the bottom of the drop-down list). The list is over a number of pages, and tells you if someone is reading a thread, staring blankly at the index, PM'ing, etc.; however, I still miss the short list at the bottom of the index.

[Nazo]

Hey, you might want to check out the new photos section of the Members tab.
The last couple photos in the first row, when clicked on, reveal some...interesting...results.

[Shevanel]

Originally Posted by Nazo

Hey, you might want to check out the new photos section of the Members tab.
The last couple photos in the first row, when clicked on, reveal some...interesting...results.

Came here to post this. lawl.

[thecat06]

Originally Posted by Shevanel

Came here to post this. lawl.

Wow...lol. I don't think that's right at all... Also if you click on the last one, it comes up something totally different.

[Chouchou]

I did the same thing, and got the same results. Usually, the page loads for a while, but the picture is missing.

[sanguine]

i laughed so hard when the picture says skydiving shirtless but pops up with a diagram of the anus/sphincter when clicked on

[J Snow]

I pointed this out in a thread in the feedback section, but I think there might have been a mistake in the message at the top of the screen. I think that stuff was only lost after 6AM CT 1/25 (instead of 1/24) because I have posts from around 2:30 AM CT 1/25. The message at the top makes it sound like there was like more than 40 hours of data lost, and it seems to me like there was closer to 20.

[Chouchou]

I'm in the Central Time zone, and I'm seeing posts from the night (about 11 PM) of 1/24.

[thecat06]

Originally Posted by Chouchou

I'm in the Central Time zone, and I'm seeing posts from the night (about 11 PM) of 1/24.

That sounds about right.. I'm in the eastern time zone so I'm seeing posts around midnight.

EDIT:I also just clicked on what are you thinking and it brought this up instead....

[Charni]

Could the problem be that a homophobic hacker hacked into the website and did hacker stuff which caused the problems?

[Tiny Catastrophe]

Originally Posted by Charni

Could the problem be that a homophobic hacker hacked into the website and did hacker stuff which caused the problems?

I was thinking the exact same thing.

[Mad Man L]

I doubt. Generally, hackers (like anonymous) rely on DDoS (Direct Denial of Service) attacks. Basically, that overloads servers. Given that firstly, EC is a nothing website (as in, if someone attacked it, you might make a few teens cry, but nothing like blacking out Facebook, tumblr or other things), I highly doubt anybody bothered hacking it. Secondly, most homophobes are old farts who fail at technology, and probably would delete system32 if told to by a troll or other internet annoyance.

The reasons supplied make sense, I've seen this on vBulletin forums before. Something screws up, and they have to do a roll back.

However, if my timezone calculation is correct, one of my posts from 6pm Central Time 25/01/12 (10am AEST on the 26th) is still up, so I guess something from that brief period that it was up was saved.

[Chip]

Nah, it's not a hack. There's a pretty complicated technical explanation for most of what happened, and that, in turn, could also explain the rest of the problems. Additionally, the servers were checked for intrusion and there was no sign of any intrusion or corruption of any files in the portion of the site where the scripts live.

And yes, there are some posts newer than 6am 1/24. We had the site up for short periods with the restored (6am 1/24) database, and then took it down again because we were still having issues, but didn't have to do another database restore. So if you happened to post after restore, but before we were back up completely, your posts would be there. It was close to 36 hours, but the site was online for a number of hours here and there, making it not a clean break.

[Charni]

What is system32?

[Chip]

If you're on a Windows machine, it's the directory where all of the system files that make the computer run are. There are various scare messages that tell you to delete files in system32 or sometimes the whole system32 directory... and Mad Man was saying that the non-computer-literate will sometimes follow the directions they get in these scare emails.

[Charni]

So if you delete anything from system32 you will stuff up your computer?

[Shebbs]

Originally Posted by Shevanel

Came here to post this. lawl.

And we all know I obviously got curious and had to try it... Lmao.

[Jonathan]

Originally Posted by Shebbs

And we all know I obviously got curious and had to try it... Lmao.

Haha, you're not the only one

[JohnnyBoy]

Originally Posted by sanguine

i laughed so hard when the picture says skydiving shirtless but pops up with a diagram of the anus/sphincter when clicked on

Yea those are my photos...I promise me skydiving without a shirt on looks nothing like a sphincter...I hope this gets fixed soon

[fiddledeedee]

Thank-you for working so hard to fix everything!