kaioa.com

Mmm... crispy sectors

It's toast

The weekend started with a premonition. Well, maybe "bad omen" is more accurate. Some rather bad SQL errors showed up, but they were sorted out quickly. The next day the site disappeared. "403 - forbidden", it said. That's just great. Did I already mention that I'm amazingly unlucky these days?

Page 403s, FTP says the password is wrong, but email works. So, I waited for the phone support hours to start. Gee. Fast forward about an hour and I got some techy to talk to. Apparently some HDD failed, it got replaced, some files were wiped (well, all of them - except for some now empty directories), and some access rights were nuked as well.

After having him fix the FTP access rights I started with the rollback. Fortunately I had a rather recent backup and only a few new files had to be located manually. An hour later the page worked fine again and I only had to upload a few other files. Invalid password. Are you kidding me?

The site also 403ed again. Not cool. Since I checked the logs right after the rollback, I knew it had been down for over 6 hours by now. I called tech support again. Their rollback script was faulty (oh really?) and since it also affects other customers they decided to restart the process. Everything should be back to normal within 1-2 hours.

"Alright", I thought after taking a take a deep breath, "I'll just wait a bit and redo all that stuff". Not much of a problem, right?

Wrong.

Somebody set up us the bomb

While thinking about what to do till then, my temporarily peace was disrupted by a police announcement. "There is a WW2 dud blabla... evacuate until 3pm blabla."

After crosschecking it with local websites and Google maps, I knew for sure that I was inside of the evacuation radius. Oh joy.

At least the weather was nice. I spent the day sitting in the sun and reading some Full Metal Panic mangas - not exactly what I had in mind, but well... not much of a choice there. FMP is pretty entertaining by the way. I like the anime a bit more though.

Unsurprisingly there wasn't an explosion. Well, the day was wasted already, but the break wasn't that bad.

Back on track

As you can probably tell the rollback went smoothly at the end. Their second attempt to fix things didn't fix anything though. There were only a few empty directories. I had to re-upload all that stuff again with my puny upstream. Then the extra files, then chmodding, clearing caches, directory passwords, etc.

All in all far more work than it should have been. And then there was also that big interruption in the middle. With all those annoyances I'm seriously behind schedule. The next couple of weeks won't be much fun, I'm afraid.

Well, sorry for the outage. And please excuse the sloppy illustration. I just wanted to draw (and vent) a bit. Maybe I should have used Inkscape's new 3D box tool for this. The barcodes are also upside down and mirrored - I'm aware of this. I'm just too lazy to fix it. ;)