It has been an incredibly long three weeks. At work, we’re getting close to relaunching our main website: I’ve spent quite a bit of my time working with Eucalyptus, Merlin, mod_passenger and all sorts of things. In addition to that, my most beloved host, eris.discordians.net, suffered an embarrassing disk failure, which resulted in exposing a number of administrative weaknesses which resulted in having backups that were two months old.
To put it succinctly: I had set up network backups, which resulted in too much bandwidth usage. I won’t comment on this because it isn’t useful; it’s not an excuse, merely more straw on the camel’s back. Anyways – I had been planning moving eris for quite a long time to an HVM on a much beefier box, ashpool. With the new release of FreeBSD this became more of a reality, and I had made a fresh copy of everything into the new HVM in preparation for the migration. I had intended to fix the backups after this, and decided having two exact, updated copies of everything was sufficient.
Then, I got busy, and a bunch of other shit happened, and when the array failed (RAID 10, two drives simultaneously, awesome), I was caught with my pants down.
A few users had interesting conspiracy theories as to why this happened; they do not need to be repeated here. Suffice to say, I have been burnt very badly by my lack of attention, and it made me feel like a huge jackass.
As a result of this, I can now say I know more about the Thunderbird and mail.app mailstores than I ever wanted to. I ended up salvaging several users’ emails from their local caches. Thunderbird is good enough to keep messages not on the server indefinitely, until the next mailbox compaction; Mail.app helpfully deletes them immediately.
Neither uses what I would call a standard format: Mail.app uses something resembling maildirs (inasmuch as every message is a file; that’s about where the similarity ends). Thunderbird, helpfully, uses a nonstandard mbox format (I should say – nonstandard beyond the numerous “standard” formats) which nothing in Ruby seemed to be able to understand. This led to me knowing a lot more about Ruby’s performance characteristics with strings, since some of the local cache files were >1GB in size.
Finally, after quite a bit of work, I had a collection of crappy Ruby scripts to copy mail from a Thunderbird store to a new maildir, and to compare two maildirs, and to get this and that, and munge this, and put this here. I was able to restore thousands of messages this way, which made me more than a little happy.
In the aftermath, I’ve finally done what I wished to do (and had been planning since March) – migrating Eris away from its own hardware, and consolidating onto one machine. This has resulted in a significant savings (Eris has never been hugely profitable, and has mostly gone between periods of +10-20% and -10-50% in terms of margins), which is cause for celebration, but there are also some huge benefits of having FreeBSD virtualized (easy upgrading, etc) which I’m pretty excited about.
I do still feel pretty stupid, especially when reading something I put on this website about eight years ago saying that any admin who didn’t have good backups in multiple locations was essentially worthless. The universe takes its time, but eventually, all words must be eaten, and all proud men must be made to kneel.