EC downtime / lost posts-PMs-blogs

Hi, everyone.

We thought it would be important to explain the problems EC has been experiencing in the last 24 hours.

Starting sometime yesterday afternoon (US central time), the forum started acting erratically. Random database errors were occurring, people were getting messages that they had posted twice in the same thread in a short time, when they had not, attempting to send PMs was causing intermittent database errors, and various other problems cropped up.

The problem was identified about 8pm CT and we started working to resolve the issue. The site was up and down during the night, and the initial fix we did (restoring to the 6pm backup from Tuesday night) did not fix the problem. We subsequently went back to the 6am backup, and that still did not solve the problem. In some cases, posts were appended to the wrong threads, creating even more confusion.

Ultimately, we found there were multiple issues that contributed to the problem, creating a "perfect storm" of problems that contributed to a big mess. It was a combination of issues with the MySQL server and its related caches that manage the database that stores the posts, combined with running a very out-of-date version of vBulletin that likely contributed to the original problems.

So... we have now upgraded to the most current revision of the major version of vBulletin we're currently running (version 3), and restored the database from 6am yesterday. We've also made some tweaks, and (for the time being at least) removed the feature that shows who is online, as, with so many users, that was causing excess load on the server that may have contributed to the problem.

We've tested everything else and believe that we've solved the problems.

What this means to you:

-- Any posts, blog entries, threads, member registrations, wall postings, PMs, or any other activity done between Tuesday 6am and 11pm Central Time tonight (i.e, Wednesday, when I'm writing this) have been lost with restoration of the backup.

-- If you registered between yesterday at 6am and now, you will need to register again.

-- If you see any odd or erratic behavior by the forum software, any error messages, etc., please report them in this thread so we can keep an eye on things and take any action required.

The staff understands how frustrating it can be to have entire messages, threads, blog posts, and so forth just disappear, particularly when you've spent a bunch of time writing them. And the staff lost all of their posts (and the admins, most of today trying to get things fixed) so it's a frustrating situation for everyone. But fortunately, these sorts of problems are pretty rare, and we don't anticipate any further problems.

We appreciate your understanding!

EDIT: Member fiddledeedee has pointed out there's still a way to see who is online, which I've reposted below. Thanks!

 

Nah, it's not a hack. There's a pretty complicated technical explanation for most of what happened, and that, in turn, could also explain the rest of the problems. Additionally, the servers were checked for intrusion and there was no sign of any intrusion or corruption of any files in the portion of the site where the scripts live.

And yes, there are some posts newer than 6am 1/24. We had the site up for short periods with the restored (6am 1/24) database, and then took it down again because we were still having issues, but didn't have to do another database restore. So if you happened to post after restore, but before we were back up completely, your posts would be there. It was close to 36 hours, but the site was online for a number of hours here and there, making it not a clean break.

 

If you're on a Windows machine, it's the directory where all of the system files that make the computer run are. There are various scare messages that tell you to delete files in system32 or sometimes the whole system32 directory... and Mad Man was saying that the non-computer-literate will sometimes follow the directions they get in these scare emails.