How to lose your entire business (and 6 years of history) in one easy step
 

Hey, all,

Just read about Journalspace on Slashdot and thought it might be a word to the wise for folks here.

Journalspace.com, a free blogging site, had some 14,000 bloggers.

Last week, they had a drive failure in their database server. They were using RAID 1 (drive mirroring) as their sole means of backup, meaning they had their server set so that anything written to one drive was mirrored to another; this theoretically allows you, in the event of a drive crash, to simply replace the bad drive and everything's fine because the data is mirrored.

Many ISPs do this and some tell their customers that it eliminates the need for individual backups.

Unfortunately, Journalspace just discovered that the problem wasn't that the drives failed, but that the data was overwritten and is completely unrecoverable. Someone or some thing caused all the data to be overwritten, and of course, since the drives were mirrored, the backup (mirror) was overwritten as well.

In other words, their entire business, and some 6 years worth of some 14,000 bloggers entries, are gone. They are simply giving up and closing down.

This should serve as a sobering reminder to double, triple, and quadruple check on your backup arrangements.

I recommend that people not trust their ISPs and make a copy of their own backups, at least several times a year, of all their sites, MySQL databases, HTML, etc. No matter whom your ISP is, I think it's a wise move to keep copies of your sites, MySQL databases, etc somewhere other than in your ISP's data center... download it to your local computer, copy it across to a server in somebody else's data center, whatever it takes.

If you rely on an IT guru (as Journalspace did) to handle your backup arrangements, I also recommend that you ask somebody else to check it out, see what's going on, and make sure that you have a bulletproof solution.

I also believe (even though NatNet tells me it's "old school" and obsolete) in keeping archival tape backups, since tapes of the right type (SDLT or LTO) are considered more stable and reliable than hard drives, and are the storage medium that the Library of Congress uses to archive crucial media.

The owner of Journalspace realizes that it's his own fault for not double-checking his IT guy's decisions, so I'm posting this in hopes that people might take steps to ensure that their backup procedures are secure to prevent something like this from happening.

You can read more here.

Welp, Google is going to be absorbing alot of those people into Blogger... I can only imagine what his face must of looked like when he got that phone call.

Wow that really sucks. I don't know that i would just give up though with that many users. Maybe the users of the site would understand that shit happens? Seems like to many loyal users to just throw it away.

Then again, what does a big greasy Italian from Brooklyn know...

Most business use DLT/LTO backup tapes, or use a remote backup that archive the data on tapes each month.

That said.. if the data was overwritten, it could still be restored by several methods. Both going after the SQL logs or simply recreating the overwritten date on the disk.

Whoever wrote that story either dont know what they are talking about, or they are quoting people who dont know what they are talking about.

PS. I would like to buy their 404 traffic :)

Darwinism at work! :thumbsup

cccc that sucks

CloneBox from bettercgi.com is what I use and suggest to others.

Off site copies of sites ready to go live whenever needed or automatically if a server goes down.

Wow, poor owners.

If you have multiple servers,then best is to set a cron on servers to do backup and cron on your local computer to download those backup so lets say if datacenter explode you will be prepared.

Very interesting story...

I do make regular backups myself manually, so if there's anyone to blame it's me lol.

But in this case with an estimate of 14k members i would keep fighting, shit happens!

Damn.... that is brutal. A good lesson for the rest of us to mind our shit.

Oh dear what a way to go!

That really sucks for them. Thing is, we see this happen all the time.
Everyone thinks it won't happen to them, or just assumes that their web
host is taking care of it. Many, like journalspace, think RAID is backup of
some sort. They learned what I've posted here several times - RAID IS
NOT BACKUP.

Here are some critical features of a proper backup system, as we learned
while designing Clonebox (which is super backup on steroids):

The backup must be on a different machine - spare copies on the same
server can be handy, but they'll likely be lost at the same time as the main copy.

Backups must be rotated, keeping several copies. Otherwise you end up like
journalspace with two copies of crap when your backup runs right AFTER the
files are deleted.

Backups must be testable and tested - far too many people find out their backups
are unreadable or otherwise unusable after it's too late.

Backups must be automatic - while people may start out keeping up with a
manual system, eventually they'll go months without a backup - right before
disaster strikes.

For web sites, backups must be immediately accessible - reloading 1 TB of
content from tape or uploading it from home would take weeks, by which time
most customers have canceled.

Alphared, the hurricanes, and similar situations have taught us that backups
must be off site - a backup provided by your host is useless when your host
is raided by police, has their connection cutoff, has a fire, etc.

Those are some things to check your current backup strategy against.
If you're not confident in your current system, Clonebox will provide rotating
backups for $5 / month, or four hot spare clones, which let you boot your
backup and have the site live in seconds for $24 / month.

it is things like this that make my prostate feel enlarged.... dam I feel for those guys...just the thought of loosing so much work makes me wanna hurl

no back up = stupid stupid stupid

Raid it however you wish, you still need back ups and back ups of the back ups.

There are only two kinds of computer users---those who have lost data and those who will lose data.

RAID ALONE IS NOT A BACKUP STRATEGY. It is a recipe for disaster.

Print what Raymor posted (#15) about backups and put it behind a refrigerator magnet.

Most businesses who post on GFY would be dead without their data. Data is a business asset ... just like money ... just like buildings ... just like traffic that converts like crazy.

Gotta run. I need to run my rotational backups and take the backup disk to my safe deposit box before the bank closes.

I do consulting for large companies and one of the I preach the most is backup / disaster recovery. If you look at statistics. Most companies who have an outage of 24 hours or more. Go out of business within a years time. If you have a site of that size, and you dont have a DR Site. With data replication. You are out of your mind.

Raid will never replace backups. Raid 1 is at its best when considered a means for protection against drive failure. Cloud is considered a new form of a worry-free semi-cluster like solution - but it requires backup in most cases.

6 years, and 14,000 bloogers - they didnt learn this? Not a very solid business plan.

Damn, that sure was a great way to start the new year.

Offline backups... #1 important process for a server

soooo underdone

This is VERY important - OC3/PR took it a step farther recently!

We just upgraded our backup solutions and are migrating customers now!


The new backup system is on a scalable redundant cluster capable of growing to 100 petabytes operating on R1soft (enterprise backup software www.r1soft.com). If a customers server dies for some reason - we can click 1 button and restore it on any hardware - after the restore is complete the server reboots and starts pinging again - our customers do not even need to reconfigure their OS or install their scripts - it backs up EVERYTHING. On top of this - it supports backing up MINUTELY! I have a customer who:

Backs up every 5 minutes and stores the restore points for 24 hours (so 300+ restore points in a day)
Backs up daily and saves restore points for a month
Backs up Monthly and saves it for a year!

Because they use incremental backups the space required for additional restore points is almost nothing.


So - BACK UP!

Miracles do happen. Modify that to if you have one or more servers and you have a winner. As I always tell people, you never know how good your backups are until you need them.

Always have your own local backups, at least.

Chains of failure
 

Something that tends to get lost in talking about data backup and recovery is the sequence of events that create the backup and events that can corrupt that process.

As in the case of the RAID setup, it did what it was supposed to do...keep a copy of the data from drive1 on drive2. The problem here was two-fold, the data was lost/overwritten and THEN the second drive simply duplicated the missing/overwritten data.

So, everyone starts talking about keeping copies on another machine, using off-site storage and other wonderful ideas. However, if the user came along right AFTER the data was overwritten and made a copy and stored it off-site they would still have a copy of nothing.

Backup is not just about multiple copies, it is about ensuring data integrity throughout the process. Not only do you have to copy the data, you have to check and make sure that it is indeed the data you want and that it can be restored in a timely manner.

There are many horror stories of IT departments that have a multiple/redundant copies of their critical data (including those wonderful tapes) and when disaster strikes they find out that all of their tapes, drive images, and other data backups are somehow unusable.

:2 cents:

Thats a big for sure. I want to see that picture. Hell, that might be enough to take me to my knees.

I dont keep tape backups, but I have 2 seperate Externals specificly for backing up my sites. The one at my home I back up every time I finish a project or pretty much anything I do. The other is in my friend Brookes house, I pull a weekly back up with that one, god forbid something should happen at either place its better to be safe then sorry.

This is a server deal though, I still feel this guy could have prevented most of the damage had he been more pro-active in his efforts. One good crash in his sites youth may have taught him the importance of protecting his work.

GREAT points, which is why we recommend customers keep at a minimum 10 restore points

Terrible way to go down

Holy shit my balls would hurt for days after that. As I was reading it, whenever someone says mirrored I always worry. A mirror of nothing is going to be nothing, like in this case. Horrible situation all around.

I recently found this out the hard way. One of my servers failed a little over a week ago and I lost 115 sites. I currently have the drive at a Data Recovery company but so far they are unable to retrieve anything. Was planning on starting 09 with a bang but this definitely set me way behind. I have been using Google cache to try and salvage what I can. Anyone know any other sites besides archive.org that may have a cache of my sites?

That REALLY sucks for JournalSpace and all of the bloggers there. In their situation they should have been far more careful in regards to backing up their stuff since they had some 14,000 people relying on their service. I just run a bunch of free sites and assumed my host had backups. As always, I tend to have to learn my lessons first hand. :disgust

You just got yourself a new customer. :thumbsup

thats what I meant by 'underdone' :thumbsup

I don;t if a should laugh or feel sorry for them....it's funny, but not.

they're like oh well, we'll just pack up and shut it the business down...the bloggers are crying...

One thing I didn't mention (and wasn't discussed in the article linked)... the site owner said in his blog that he immediately sent the discs to drivesavers, which is one of the better data recovery people, and they were the ones that determined the data could not be recovered.

Given that it is (theoretically at least) possible to recover data even after the drive has been overwritten, I have to assume that either it was too expensive, or not practical, or Drivesavers wasn't equipped with that technology.

All of the suggestions in this thread are good ones. I would trust Raymor's solutions... he's never steered us wrong... and for us, we rely on NatNet's backups but also keep most everything backed up locally on one of our RAID-5 arrays here... and the arrays themselves are backed up onto SDLT.

they could make mad loot with all that 404 traffic

May have to give you guys a try...

oh shit... talk about a nightmare :/

Google Cache is my backup plan. :thumbsup

I was going to suggest they could probably restore a good chunk of entries via archive.org (even if they just turned it into a historical archive), but they're following the "accept some, deny all" robots.txt model which means that archive.org isn't allowed to crawl the site. If that block has been in place for some time then there's no chance of any recent posts being in the archive.

User-agent: *
Disallow: /
User-agent: kinja-imagebot
Disallow:
User-agent: kinjabot
Disallow:
User-agent: Mediapartners-Google*
Disallow:
User-agent: Googlebot
Disallow:
User-agent: msnbot
Disallow:

Hmm... Google, Google Adsense, MSN/Live, and kinja are the only bots permitted to crawl the site.

Where's Yahoo? Blocked......

How much do you charge for that service? or is it included for free with any server?

:) Ali days that all the time.
I guess you could say there are three kinds of computer users,
those who have lost data, those who will data, and those will
really solid backup procedures. (Though even with really
solid procedures you could well use a few hours.)

Thanks for that - I'm glad you found it to be solid advice.
We too use a safe deposit box as one part of an overall back up system.
It's good for protecting data in the office in case of fire, burglary, etc.
You're probably not going to go to the safe deposit every day, so we
also use separate daily backups, and restoring from removable media
is a lot slower than simply booting the hot clone, so the disk in the
safe deposit box isn't a complete solution at all, but it's a good plan C
or plan D to avoid losing everything completely in the case where the
primary back up fails for some reason - which seems to happen more
often than one would expect, simply because dumping hundreds of
gigabytes of data is very hard on any media - tape, HD, or whatever.

If put hard work in building up a blog and you lost it because of the provider's mistake, would you trust them again ? I wouldn't.

Been there, done that.

No chance :2 cents:

holy shit wouldn't you be be pissed :(

good advise in your post there gaybucks

The shit hit the fan on that one!!