After two weeks and two days (not that I was counting) of server downtime, we’re back to normal. What happened? I was doing some regular maintenance on the system (updating config files for updated software), and managed to completely break our authentication scheme. That means that nobody could authenticate to the server, and thus nobody could really do anything to modify the server. Talk about security!

The guys over at Waveform, our colocation provider, were pretty great in helping us get back online, even after I nagged the hell out of them. Thanks, Waveform!

I also put in a few secure backdoors, so this type of thing will never happen again. My apologies for all of the server’s users and visitors.