The last 48 hours haven’t been fun. Technology is a fantastic thing, but it’s never perfect. When if fails, I’m not sure there’s really that much preparation you can have… but you do have to react.
You may have noticed that our site was getting gruelingly slow the last couple weeks. It was strange given the fact that we have it on a great hosting package combined with a database server and content delivery network. Since we had a lot of space, we hosted other sites there, too… and that was our mistake!
One of our projects is a social media monitoring tool that connects to Twitter and Facebook, collecting data on thousands of teams in the sports market. Once a day it gathers fan and follower information, collecting it in the database. We’ve been doing a lot of development on the project and recently noticed that some of the stats were wrong. Our client, Pat Coyle, has been patient with us as we’ve been troubleshooting the issue.
Then all h**l broke loose! It appears the process for gathering the info started running within minutes instead of once a day. Our database grew to over 1G within days, slowing our server and taking tons of space on it. The other night I was actually watching as each site we had on the account started going down one by one. Ugh.
We were already making plans to move Martech over to WPEngine to put it in a dedicated environment with backups, integrated content delivery, and a blazing fast set of servers. We have a couple other clients on it and have been extremely happy with both the service and their incredible support. It’s not that Mediatemple was bad, it’s simply that this environment was built for publication blogs like ours that get a ton of traffic. In the middle of the night, I wrote the guys at WPEngine and they had me up by the morning! Thanks guys!
Next, we started looking at how to fix the database. It actually halted the database server and corrupted the largest table (the one with ALL the central data!). Since the server was full, we couldn’t do repair… we couldn’t access the files, couldn’t back it up… we were stuck. The folks at MediaTemple jumped in and repaired the table. We were then able to make a full backup and begin bringing the other sites back up.
The move to WPEngine wasn’t without pain. Since we couldn’t access our database, we had to take a recent snapshot of the database… which for some reason lost all of our category alignment in the process. We have off-site WordPress backups, too, but our database is so large that putting together all the backup segments was going to take way too long.
So, we restored the data and been rooting through the 2,500+ posts and carefully recategorizing them. I’m sure we’re going to take a bit of a hit on SEO because that changed URL paths… so we took an even bigger hit and changed our permalink structure (without the category). It’s something I’ve needed to do for a while, so now was a better time than later.
We scrapped our old theme. It was graphics heavy (without CSS sprites) and wasn’t too friendly to resizing. We’ve decided just to highly modify the Twenty Eleven theme that’s standard with WordPress for now. It’s HTML5 ready and has a ton of responsive design features that were good to take advantage of.
In the meantime, Jenn held down the fort at Highbridge – juggling a few projects and knocking them out big time. Stephen pulled an all-dayer (he already works nights!), good friend Adam Small kicked in and helped, MediaTemple knocked it out of the park, and WPEngine helped, too. Thanks to everyone… we’re back to blogging again!
Now it’s time for me to get some sleep :). Then we’ll fix our iPad and mobile themes!
Doug,
I’m very sorry to hear you have had these problems. This is where an IT professional, such as myself, could have and SHOULD have been monitoring your systems to prevent such a situation. I’m sure you are getting many such comments and posts, but the situation is simple. Proper maintenance, growth planing and forecasting should have prepared your sites and your database for this level of traffic. I would be interested in speaking with you, regarding this situation, and steps that can be taken to prevent in the future. We have spoken in the past, you know me on your twitter stream as @indyscompugeek.
Daniel, HeadGeek of Indy’s Computer Geek
No doubt, Daniel! In this case, we got exactly what we paid for… and were paid for. We’re on a shoestring budget on this one.
Doug – UGH! is right. I was on this site about a week or so ago reading a blog and I did notice how slow it was. I seriously contemplated emailing you and saying thus, but I thought who am I to tell ‘The Man” that his site was “sluggish.” Now I know why! So, glad you (and your posse) were able to get it back up and functional. Have you ever considered integrating the StudioPress Genesis Framework – http://www.studiopress.com – I use if for my site and all of my client sites. Very solid piece of software – Love it!
Hi Greg! I’ve heard great things about the Genesis framework. We haven’t used it yet but I think I may get out WordPress guru, Stephen, to build a theme with it. This theme started with Twenty Eleven but it’s been totally customized and optimized. So far, it seems to be holding up well!