Hey, all,
Just read about Journalspace on Slashdot and thought it might be a word to the wise for folks here.
Journalspace.com, a free blogging site, had some 14,000 bloggers.
Last week, they had a drive failure in their database server. They were using RAID 1 (drive mirroring) as their sole means of backup, meaning they had their server set so that anything written to one drive was mirrored to another; this theoretically allows you, in the event of a drive crash, to simply replace the bad drive and everything's fine because the data is mirrored.
Many ISPs do this and some tell their customers that it eliminates the need for individual backups.
Unfortunately, Journalspace just discovered that the problem wasn't that the drives failed, but that the data was
overwritten and is completely unrecoverable. Someone or some thing caused all the data to be overwritten, and of course, since the drives were mirrored, the backup (mirror) was overwritten as well.
In other words, their entire business, and some 6 years worth of some 14,000 bloggers entries, are gone. They are simply giving up and closing down.
This should serve as a sobering reminder to double, triple, and quadruple check on your backup arrangements.
I recommend that people not trust their ISPs and make a copy of their own backups, at least several times a year, of all their sites, MySQL databases, HTML, etc. No matter whom your ISP is, I think it's a wise move to keep copies of your sites, MySQL databases, etc somewhere other than in your ISP's data center... download it to your local computer, copy it across to a server in somebody else's data center, whatever it takes.
If you rely on an IT guru (as Journalspace did) to handle your backup arrangements, I also recommend that you ask somebody else to check it out, see what's going on, and make sure that you have a bulletproof solution.
I also believe (even though NatNet tells me it's "old school" and obsolete) in keeping archival tape backups, since tapes of the right type (SDLT or LTO) are considered more stable and reliable than hard drives, and are the storage medium that the Library of Congress uses to archive crucial media.
The owner of Journalspace realizes that it's his own fault for not double-checking his IT guy's decisions, so I'm posting this in hopes that people might take steps to ensure that their backup procedures are secure to prevent something like this from happening.
You can read more
here.