When Disaster Strikes!

The last 48 hours haven’t been fun. Technology is a fantastic thing, but it’s never perfect. When if fails, I’m not sure there’s really that much preparation you can have… but you do have to react.

You may have noticed that our site was getting gruelingly slow the last couple weeks. It was strange given the fact that we have it on a great hosting package combined with a database server and content delivery network. Since we had a lot of space, we hosted other sites there, too… and that was our mistake!

One of our projects is a social media monitoring tool that connects to Twitter and Facebook, collecting data on thousands of teams in the sports market. Once a day it gathers fan and follower information, collecting it in the database. We’ve been doing a lot of development on the project and recently noticed that some of the stats were wrong. Our client, Pat Coyle, has been patient with us as we’ve been troubleshooting the issue.

Then all h**l broke loose! It appears the process for gathering the info started running within minutes instead of once a day. Our database grew to over 1G within days, slowing our server and taking tons of space on it. The other night I was actually watching as each site we had on the account started going down one by one. Ugh.

We were already making plans to move Martech over to WPEngine to put it in a dedicated environment with backups, integrated content delivery, and a blazing fast set of servers. We have a couple other clients on it and have been extremely happy with both the service and their incredible support. It’s not that Mediatemple was bad, it’s simply that this environment was built for publication blogs like ours that get a ton of traffic. In the middle of the night, I wrote the guys at WPEngine and they had me up by the morning! Thanks guys!

Next, we started looking at how to fix the database. It actually halted the database server and corrupted the largest table (the one with ALL the central data!). Since the server was full, we couldn’t do repair… we couldn’t access the files, couldn’t back it up… we were stuck. The folks at MediaTemple jumped in and repaired the table. We were then able to make a full backup and begin bringing the other sites back up.

The move to WPEngine wasn’t without pain. Since we couldn’t access our database, we had to take a recent snapshot of the database… which for some reason lost all of our category alignment in the process. We have off-site WordPress backups, too, but our database is so large that putting together all the backup segments was going to take way too long.

So, we restored the data and been rooting through the 2,500+ posts and carefully recategorizing them. I’m sure we’re going to take a bit of a hit on SEO because that changed URL paths… so we took an even bigger hit and changed our permalink structure (without the category). It’s something I’ve needed to do for a while, so now was a better time than later.

We scrapped our old theme. It was graphics heavy (without CSS sprites) and wasn’t too friendly to resizing. We’ve decided just to highly modify the Twenty Eleven theme that’s standard with WordPress for now. It’s HTML5 ready and has a ton of responsive design features that were good to take advantage of.

In the meantime, Jenn held down the fort at DK New Media – juggling a few projects and knocking them out big time. Stephen pulled an all-dayer (he already works nights!), good friend Adam Small kicked in and helped, MediaTemple knocked it out of the park, and WPEngine helped, too. Thanks to everyone… we’re back to blogging again!

Now it’s time for me to get some sleep :). Then we’ll fix our iPad and mobile themes!

Exit mobile version