Downtime Explanation: It Might Not Be Over Yet

So by now I am sure all of you have noticed that ZI has been down a lot lately. Mysql errors galore with some vgrc url stuff and all that jazz. Well, the short version is that somehow the back end software that actually allows everything to run on the server has been corrupted. We’re not sure how far the corruption goes but we do know it’s not just limited to Zelda Informer. The 2nd server of ours, the one with Metroid Wiki on it, crashed and burned a couple weeks back as well. So whatever the issues is it’s universal, and thus must be a problem we overlooked when we initially set up the servers.

So why haven’t we wiped the servers and reinstalled yet? Well the answer is simple: We are suspended from our back end rackspace access for being behind on bills. No, we don’t need your help (haha, what a change!) but what we do need is time, time that, as you can see, isn’t agreeing with the server corruption level. We will be all caught up by next friday, and should things go as planned the servers will have fresh installs of everything and we’ll be all good to go with no bills left to pay. However that doesn’t mean the downtime is over with…

The downtime is happening because of hte server corruption, and up until tonight the corruption caused a very slow connection to the backend we do have access to outside of rackspace. This meant to restart the mysql so it would work again (all be it for only a few hours before crashing again) it would take upwords of 30 minutes, assuming it didn’t time out at any point. It usually did time out, and this is why you saw so much down time even before yesterday through today. Previously we could just reboot the server from the backend on rackspace, but the problem continued to get worse and worse.

Good news is the site is online right now, and the WHM backend is working at normal speeds for the first time in weeks. It’s sort of odd really because we didn’t actually do anything: It appears to have “auto corrected”. I’m not sure, but were online… for now. I wouldn’t count on us being stable until some time next week, but if this site stays online for the next 24 to 48 hours things may be back to normal after all. We are postponing all article releases until we are sure the server is stable enough to be online for you to read them. Assuming the server is still up and running after I sleep, we’ll resume the news reporting aspect per usual.

I apologize for the downtime and we are working towards a permanent resolution to ensure this wont happen again, Bad timing bit us it the ass yet again, but we’ll survive. Rest assured folks, it’s going to take more than some corrupted servers to keep ZI down. Don’t worry, when we declare ourselves stable we’ll have a nice big celebration of awesomeness. You’ll have to wait and see what that entails!