How to lose your entire business (and 6 years of history) in one easy step - GoFuckYourself.com

Gaybucks · 01-03-2009, 06:21 AM

Hey, all,

Just read about Journalspace on Slashdot and thought it might be a word to the wise for folks here.

Journalspace.com, a free blogging site, had some 14,000 bloggers.

Last week, they had a drive failure in their database server. They were using RAID 1 (drive mirroring) as their sole means of backup, meaning they had their server set so that anything written to one drive was mirrored to another; this theoretically allows you, in the event of a drive crash, to simply replace the bad drive and everything's fine because the data is mirrored.

Many ISPs do this and some tell their customers that it eliminates the need for individual backups.

Unfortunately, Journalspace just discovered that the problem wasn't that the drives failed, but that the data was overwritten and is completely unrecoverable. Someone or some thing caused all the data to be overwritten, and of course, since the drives were mirrored, the backup (mirror) was overwritten as well.

In other words, their entire business, and some 6 years worth of some 14,000 bloggers entries, are gone. They are simply giving up and closing down.

This should serve as a sobering reminder to double, triple, and quadruple check on your backup arrangements.

I recommend that people not trust their ISPs and make a copy of their own backups, at least several times a year, of all their sites, MySQL databases, HTML, etc. No matter whom your ISP is, I think it's a wise move to keep copies of your sites, MySQL databases, etc somewhere other than in your ISP's data center... download it to your local computer, copy it across to a server in somebody else's data center, whatever it takes.

If you rely on an IT guru (as Journalspace did) to handle your backup arrangements, I also recommend that you ask somebody else to check it out, see what's going on, and make sure that you have a bulletproof solution.

I also believe (even though NatNet tells me it's "old school" and obsolete) in keeping archival tape backups, since tapes of the right type (SDLT or LTO) are considered more stable and reliable than hard drives, and are the storage medium that the Library of Congress uses to archive crucial media.

The owner of Journalspace realizes that it's his own fault for not double-checking his IT guy's decisions, so I'm posting this in hopes that people might take steps to ensure that their backup procedures are secure to prevent something like this from happening.

You can read more here.

Iron Fist · 01-03-2009, 06:24 AM

Welp, Google is going to be absorbing alot of those people into Blogger... I can only imagine what his face must of looked like when he got that phone call.

Carmine Raguso · 01-03-2009, 06:25 AM

Wow that really sucks. I don't know that i would just give up though with that many users. Maybe the users of the site would understand that shit happens? Seems like to many loyal users to just throw it away.

Then again, what does a big greasy Italian from Brooklyn know...

Machete_ · 01-03-2009, 06:27 AM

Most business use DLT/LTO backup tapes, or use a remote backup that archive the data on tapes each month.

That said.. if the data was overwritten, it could still be restored by several methods. Both going after the SQL logs or simply recreating the overwritten date on the disk.

Whoever wrote that story either dont know what they are talking about, or they are quoting people who dont know what they are talking about.

Machete_ · 01-03-2009, 06:28 AM

PS. I would like to buy their 404 traffic

DamageX · 01-03-2009, 06:56 AM

Darwinism at work!

seeandsee · 01-03-2009, 07:40 AM

cccc that sucks

TisMe · 01-03-2009, 07:45 AM

CloneBox from bettercgi.com is what I use and suggest to others.

Off site copies of sites ready to go live whenever needed or automatically if a server goes down.

The Duck · 01-03-2009, 07:47 AM

Wow, poor owners.

Klen · 01-03-2009, 07:54 AM

If you have multiple servers,then best is to set a cron on servers to do backup and cron on your local computer to download those backup so lets say if datacenter explode you will be prepared.

DraX · 01-03-2009, 08:31 AM

Very interesting story...

I do make regular backups myself manually, so if there's anyone to blame it's me lol.

But in this case with an estimate of 14k members i would keep fighting, shit happens!

cykoe6 · 01-03-2009, 09:02 AM

Damn.... that is brutal. A good lesson for the rest of us to mind our shit.

Si · 01-03-2009, 09:04 AM

Oh dear what a way to go!

raymor · 01-03-2009, 09:46 AM

That really sucks for them. Thing is, we see this happen all the time.
Everyone thinks it won't happen to them, or just assumes that their web
host is taking care of it. Many, like journalspace, think RAID is backup of
some sort. They learned what I've posted here several times - RAID IS
NOT BACKUP.

Here are some critical features of a proper backup system, as we learned
while designing Clonebox (which is super backup on steroids):

The backup must be on a different machine - spare copies on the same
server can be handy, but they'll likely be lost at the same time as the main copy.

Backups must be rotated, keeping several copies. Otherwise you end up like
journalspace with two copies of crap when your backup runs right AFTER the
files are deleted.

Backups must be testable and tested - far too many people find out their backups
are unreadable or otherwise unusable after it's too late.

Backups must be automatic - while people may start out keeping up with a
manual system, eventually they'll go months without a backup - right before
disaster strikes.

For web sites, backups must be immediately accessible - reloading 1 TB of
content from tape or uploading it from home would take weeks, by which time
most customers have canceled.

Alphared, the hurricanes, and similar situations have taught us that backups
must be off site - a backup provided by your host is useless when your host
is raided by police, has their connection cutoff, has a fire, etc.

Those are some things to check your current backup strategy against.
If you're not confident in your current system, Clonebox will provide rotating
backups for $5 / month, or four hot spare clones, which let you boot your
backup and have the site live in seconds for $24 / month.

qxm · 01-03-2009, 09:58 AM

it is things like this that make my prostate feel enlarged.... dam I feel for those guys...just the thought of loosing so much work makes me wanna hurl

StuartD · 01-03-2009, 10:00 AM

no back up = stupid stupid stupid

Raid it however you wish, you still need back ups and back ups of the back ups.

PhantomFrog · 01-03-2009, 11:07 AM

There are only two kinds of computer users---those who have lost data and those who will lose data.

RAID ALONE IS NOT A BACKUP STRATEGY. It is a recipe for disaster.

Print what Raymor posted (#15) about backups and put it behind a refrigerator magnet.

Most businesses who post on GFY would be dead without their data. Data is a business asset ... just like money ... just like buildings ... just like traffic that converts like crazy.

Gotta run. I need to run my rotational backups and take the backup disk to my safe deposit box before the bank closes.

Supz · 01-03-2009, 11:12 AM

I do consulting for large companies and one of the I preach the most is backup / disaster recovery. If you look at statistics. Most companies who have an outage of 24 hours or more. Go out of business within a years time. If you have a site of that size, and you dont have a DR Site. With data replication. You are out of your mind.

skrinkladoo · 01-03-2009, 01:58 PM

Raid will never replace backups. Raid 1 is at its best when considered a means for protection against drive failure. Cloud is considered a new form of a worry-free semi-cluster like solution - but it requires backup in most cases.

6 years, and 14,000 bloogers - they didnt learn this? Not a very solid business plan.

Miguel T · 01-03-2009, 02:00 PM

Damn, that sure was a great way to start the new year.

CrkMStanz · 01-03-2009, 02:00 PM

Offline backups... #1 important process for a server

soooo underdone

PR|Jordan · 01-03-2009, 02:10 PM

This is VERY important - OC3/PR took it a step farther recently!

We just upgraded our backup solutions and are migrating customers now!

The new backup system is on a scalable redundant cluster capable of growing to 100 petabytes operating on R1soft (enterprise backup software www.r1soft.com). If a customers server dies for some reason - we can click 1 button and restore it on any hardware - after the restore is complete the server reboots and starts pinging again - our customers do not even need to reconfigure their OS or install their scripts - it backs up EVERYTHING. On top of this - it supports backing up MINUTELY! I have a customer who:

Backs up every 5 minutes and stores the restore points for 24 hours (so 300+ restore points in a day)
Backs up daily and saves restore points for a month
Backs up Monthly and saves it for a year!

Because they use incremental backups the space required for additional restore points is almost nothing.

So - BACK UP!

baddog · 01-03-2009, 02:12 PM

Quote:

Originally Posted by KlenTelaris

If you have multiple servers,then best is to set a cron on servers to do backup and cron on your local computer to download those backup so lets say if datacenter explode you will be prepared.

Miracles do happen. Modify that to if you have one or more servers and you have a winner. As I always tell people, you never know how good your backups are until you need them.

Always have your own local backups, at least.

RevTKS69 · 01-03-2009, 02:17 PM

Something that tends to get lost in talking about data backup and recovery is the sequence of events that create the backup and events that can corrupt that process.

As in the case of the RAID setup, it did what it was supposed to do...keep a copy of the data from drive1 on drive2. The problem here was two-fold, the data was lost/overwritten and THEN the second drive simply duplicated the missing/overwritten data.

So, everyone starts talking about keeping copies on another machine, using off-site storage and other wonderful ideas. However, if the user came along right AFTER the data was overwritten and made a copy and stored it off-site they would still have a copy of nothing.

Backup is not just about multiple copies, it is about ensuring data integrity throughout the process. Not only do you have to copy the data, you have to check and make sure that it is indeed the data you want and that it can be restored in a timely manner.

There are many horror stories of IT departments that have a multiple/redundant copies of their critical data (including those wonderful tapes) and when disaster strikes they find out that all of their tapes, drive images, and other data backups are somehow unusable.

jmcb420 · 01-03-2009, 02:21 PM

Quote:

Originally Posted by sharphead

I can only imagine what his face must of looked like when he got that phone call.

Thats a big for sure. I want to see that picture. Hell, that might be enough to take me to my knees.

I dont keep tape backups, but I have 2 seperate Externals specificly for backing up my sites. The one at my home I back up every time I finish a project or pretty much anything I do. The other is in my friend Brookes house, I pull a weekly back up with that one, god forbid something should happen at either place its better to be safe then sorry.

This is a server deal though, I still feel this guy could have prevented most of the damage had he been more pro-active in his efforts. One good crash in his sites youth may have taught him the importance of protecting his work.

PR|Jordan · 01-03-2009, 02:37 PM

Quote:

Originally Posted by RevTKS69

Something that tends to get lost in talking about data backup and recovery is the sequence of events that create the backup and events that can corrupt that process.

As in the case of the RAID setup, it did what it was supposed to do...keep a copy of the data from drive1 on drive2. The problem here was two-fold, the data was lost/overwritten and THEN the second drive simply duplicated the missing/overwritten data.

So, everyone starts talking about keeping copies on another machine, using off-site storage and other wonderful ideas. However, if the user came along right AFTER the data was overwritten and made a copy and stored it off-site they would still have a copy of nothing.

Backup is not just about multiple copies, it is about ensuring data integrity throughout the process. Not only do you have to copy the data, you have to check and make sure that it is indeed the data you want and that it can be restored in a timely manner.

There are many horror stories of IT departments that have a multiple/redundant copies of their critical data (including those wonderful tapes) and when disaster strikes they find out that all of their tapes, drive images, and other data backups are somehow unusable.

GREAT points, which is why we recommend customers keep at a minimum 10 restore points

Drake · 01-03-2009, 02:45 PM

Terrible way to go down

hypedough · 01-03-2009, 02:52 PM

Holy shit my balls would hurt for days after that. As I was reading it, whenever someone says mirrored I always worry. A mirror of nothing is going to be nothing, like in this case. Horrible situation all around.

st0ned · 01-03-2009, 03:37 PM

I recently found this out the hard way. One of my servers failed a little over a week ago and I lost 115 sites. I currently have the drive at a Data Recovery company but so far they are unable to retrieve anything. Was planning on starting 09 with a bang but this definitely set me way behind. I have been using Google cache to try and salvage what I can. Anyone know any other sites besides archive.org that may have a cache of my sites?

That REALLY sucks for JournalSpace and all of the bloggers there. In their situation they should have been far more careful in regards to backing up their stuff since they had some 14,000 people relying on their service. I just run a bunch of free sites and assumed my host had backups. As always, I tend to have to learn my lessons first hand.

st0ned · 01-03-2009, 03:40 PM

Quote:

Originally Posted by raymor

That really sucks for them. Thing is, we see this happen all the time.
Everyone thinks it won't happen to them, or just assumes that their web
host is taking care of it. Many, like journalspace, think RAID is backup of
some sort. They learned what I've posted here several times - RAID IS
NOT BACKUP.

Here are some critical features of a proper backup system, as we learned
while designing Clonebox (which is super backup on steroids):

The backup must be on a different machine - spare copies on the same
server can be handy, but they'll likely be lost at the same time as the main copy.

Backups must be rotated, keeping several copies. Otherwise you end up like
journalspace with two copies of crap when your backup runs right AFTER the
files are deleted.

Backups must be testable and tested - far too many people find out their backups
are unreadable or otherwise unusable after it's too late.

Backups must be automatic - while people may start out keeping up with a
manual system, eventually they'll go months without a backup - right before
disaster strikes.

For web sites, backups must be immediately accessible - reloading 1 TB of
content from tape or uploading it from home would take weeks, by which time
most customers have canceled.

Alphared, the hurricanes, and similar situations have taught us that backups
must be off site - a backup provided by your host is useless when your host
is raided by police, has their connection cutoff, has a fire, etc.

Those are some things to check your current backup strategy against.
If you're not confident in your current system, Clonebox will provide rotating
backups for $5 / month, or four hot spare clones, which let you boot your
backup and have the site live in seconds for $24 / month.

You just got yourself a new customer.

CrkMStanz · 01-03-2009, 03:44 PM

Quote:

Originally Posted by RevTKS69

Something that tends to get lost in talking about data backup and recovery is the sequence of events that create the backup and events that can corrupt that process.

As in the case of the RAID setup, it did what it was supposed to do...keep a copy of the data from drive1 on drive2. The problem here was two-fold, the data was lost/overwritten and THEN the second drive simply duplicated the missing/overwritten data.

So, everyone starts talking about keeping copies on another machine, using off-site storage and other wonderful ideas. However, if the user came along right AFTER the data was overwritten and made a copy and stored it off-site they would still have a copy of nothing.

Backup is not just about multiple copies, it is about ensuring data integrity throughout the process. Not only do you have to copy the data, you have to check and make sure that it is indeed the data you want and that it can be restored in a timely manner.

There are many horror stories of IT departments that have a multiple/redundant copies of their critical data (including those wonderful tapes) and when disaster strikes they find out that all of their tapes, drive images, and other data backups are somehow unusable.

thats what I meant by 'underdone'

TyroneGoldberg · 01-03-2009, 04:16 PM

I don;t if a should laugh or feel sorry for them....it's funny, but not.

they're like oh well, we'll just pack up and shut it the business down...the bloggers are crying...

Gaybucks · 01-03-2009, 04:35 PM

One thing I didn't mention (and wasn't discussed in the article linked)... the site owner said in his blog that he immediately sent the discs to drivesavers, which is one of the better data recovery people, and they were the ones that determined the data could not be recovered.

Given that it is (theoretically at least) possible to recover data even after the drive has been overwritten, I have to assume that either it was too expensive, or not practical, or Drivesavers wasn't equipped with that technology.

All of the suggestions in this thread are good ones. I would trust Raymor's solutions... he's never steered us wrong... and for us, we rely on NatNet's backups but also keep most everything backed up locally on one of our RAID-5 arrays here... and the arrays themselves are backed up onto SDLT.

spacedog · 01-03-2009, 04:36 PM

they could make mad loot with all that 404 traffic

Ethersync · 01-03-2009, 04:57 PM

Quote:

Originally Posted by PR|Jordan

This is VERY important - OC3/PR took it a step farther recently!

We just upgraded our backup solutions and are migrating customers now!

The new backup system is on a scalable redundant cluster capable of growing to 100 petabytes operating on R1soft (enterprise backup software www.r1soft.com). If a customers server dies for some reason - we can click 1 button and restore it on any hardware - after the restore is complete the server reboots and starts pinging again - our customers do not even need to reconfigure their OS or install their scripts - it backs up EVERYTHING. On top of this - it supports backing up MINUTELY! I have a customer who:

Backs up every 5 minutes and stores the restore points for 24 hours (so 300+ restore points in a day)
Backs up daily and saves restore points for a month
Backs up Monthly and saves it for a year!

Because they use incremental backups the space required for additional restore points is almost nothing.

So - BACK UP!

May have to give you guys a try...

JD · 01-03-2009, 05:20 PM

oh shit... talk about a nightmare :/

Iron Fist · 01-03-2009, 05:32 PM

Google Cache is my backup plan.

rowan · 01-03-2009, 05:42 PM

I was going to suggest they could probably restore a good chunk of entries via archive.org (even if they just turned it into a historical archive), but they're following the "accept some, deny all" robots.txt model which means that archive.org isn't allowed to crawl the site. If that block has been in place for some time then there's no chance of any recent posts being in the archive.

User-agent: *
Disallow: /
User-agent: kinja-imagebot
Disallow:
User-agent: kinjabot
Disallow:
User-agent: Mediapartners-Google*
Disallow:
User-agent: Googlebot
Disallow:
User-agent: msnbot
Disallow:

Hmm... Google, Google Adsense, MSN/Live, and kinja are the only bots permitted to crawl the site.

Where's Yahoo? Blocked......

woj · 01-03-2009, 05:48 PM

Quote:

Originally Posted by PR|Jordan

This is VERY important - OC3/PR took it a step farther recently!

We just upgraded our backup solutions and are migrating customers now!

The new backup system is on a scalable redundant cluster capable of growing to 100 petabytes operating on R1soft (enterprise backup software www.r1soft.com). If a customers server dies for some reason - we can click 1 button and restore it on any hardware - after the restore is complete the server reboots and starts pinging again - our customers do not even need to reconfigure their OS or install their scripts - it backs up EVERYTHING. On top of this - it supports backing up MINUTELY! I have a customer who:

Backs up every 5 minutes and stores the restore points for 24 hours (so 300+ restore points in a day)
Backs up daily and saves restore points for a month
Backs up Monthly and saves it for a year!

Because they use incremental backups the space required for additional restore points is almost nothing.

So - BACK UP!

How much do you charge for that service? or is it included for free with any server?

raymor · 01-03-2009, 08:33 PM

Quote:

Originally Posted by PhantomFrog

There are only two kinds of computer users---those who have lost data and those who will lose data.

Ali days that all the time.
I guess you could say there are three kinds of computer users,
those who have lost data, those who will data, and those will
really solid backup procedures. (Though even with really
solid procedures you could well use a few hours.)

Quote:

Originally Posted by PhantomFrog

Print what Raymor posted (#15) about backups and put it behind a refrigerator magnet.
..
need to run my rotational backups and take the backup disk to my safe deposit box before the bank closes.

Thanks for that - I'm glad you found it to be solid advice.
We too use a safe deposit box as one part of an overall back up system.
It's good for protecting data in the office in case of fire, burglary, etc.
You're probably not going to go to the safe deposit every day, so we
also use separate daily backups, and restoring from removable media
is a lot slower than simply booting the hot clone, so the disk in the
safe deposit box isn't a complete solution at all, but it's a good plan C
or plan D to avoid losing everything completely in the case where the
primary back up fails for some reason - which seems to happen more
often than one would expect, simply because dumping hundreds of
gigabytes of data is very hard on any media - tape, HD, or whatever.

Doctor Dre · 01-03-2009, 09:42 PM

Quote:

Originally Posted by Carmine Raguso

Wow that really sucks. I don't know that i would just give up though with that many users. Maybe the users of the site would understand that shit happens? Seems like to many loyal users to just throw it away.

Then again, what does a big greasy Italian from Brooklyn know...

If put hard work in building up a blog and you lost it because of the provider's mistake, would you trust them again ? I wouldn't.

Jakez · 01-04-2009, 01:54 AM

Been there, done that.

Ethersync · 01-04-2009, 02:48 PM

Quote:

Originally Posted by Doctor Dre

If put hard work in building up a blog and you lost it because of the provider's mistake, would you trust them again ? I wouldn't.

No chance

papill0n · 01-04-2009, 04:04 PM

holy shit wouldn't you be be pissed

good advise in your post there gaybucks

Redrob · 01-04-2009, 04:10 PM

The shit hit the fan on that one!!

01-03-2009, 06:24 AM	#2
Iron Fist Too lazy to set a custom title Join Date: Dec 2006 Posts: 23,400	Welp, Google is going to be absorbing alot of those people into Blogger... I can only imagine what his face must of looked like when he got that phone call. __________________ i like waffles

01-03-2009, 06:56 AM	#6
DamageX Marketing & Strategy Industry Role: Join Date: Jun 2001 Location: Former nomad Posts: 14,293	Darwinism at work! __________________ Whitehat is for chumps If you don't do it, somebody else will - true story!

01-03-2009, 07:40 AM	#7
seeandsee Check SIG! Industry Role: Join Date: Mar 2006 Location: Europe (Skype: gojkoas) Posts: 50,945	cccc that sucks __________________ BUY MY SIG - 50$/Year Contact here

01-03-2009, 07:47 AM	#9
The Duck Adult Content Provider Industry Role: Join Date: May 2005 Location: Europe Posts: 18,243	Wow, poor owners. __________________ Skype Horusmaia ICQ 41555245 Email [email protected]

01-03-2009, 09:02 AM	#12
cykoe6 Confirmed User Industry Role: Join Date: Apr 2005 Location: Vegas Posts: 4,499	Damn.... that is brutal. A good lesson for the rest of us to mind our shit. __________________ бабки, шлюхи, сила

01-03-2009, 06:21 AM	#1
Gaybucks Confirmed User Join Date: May 2004 Location: Sacramento, CA Posts: 451	How to lose your entire business (and 6 years of history) in one easy step Hey, all, Just read about Journalspace on Slashdot and thought it might be a word to the wise for folks here. Journalspace.com, a free blogging site, had some 14,000 bloggers. Last week, they had a drive failure in their database server. They were using RAID 1 (drive mirroring) as their sole means of backup, meaning they had their server set so that anything written to one drive was mirrored to another; this theoretically allows you, in the event of a drive crash, to simply replace the bad drive and everything's fine because the data is mirrored. Many ISPs do this and some tell their customers that it eliminates the need for individual backups. Unfortunately, Journalspace just discovered that the problem wasn't that the drives failed, but that the data was overwritten and is completely unrecoverable. Someone or some thing caused all the data to be overwritten, and of course, since the drives were mirrored, the backup (mirror) was overwritten as well. In other words, their entire business, and some 6 years worth of some 14,000 bloggers entries, are gone. They are simply giving up and closing down. This should serve as a sobering reminder to double, triple, and quadruple check on your backup arrangements. I recommend that people not trust their ISPs and make a copy of their own backups, at least several times a year, of all their sites, MySQL databases, HTML, etc. No matter whom your ISP is, I think it's a wise move to keep copies of your sites, MySQL databases, etc somewhere other than in your ISP's data center... download it to your local computer, copy it across to a server in somebody else's data center, whatever it takes. If you rely on an IT guru (as Journalspace did) to handle your backup arrangements, I also recommend that you ask somebody else to check it out, see what's going on, and make sure that you have a bulletproof solution. I also believe (even though NatNet tells me it's "old school" and obsolete) in keeping archival tape backups, since tapes of the right type (SDLT or LTO) are considered more stable and reliable than hard drives, and are the storage medium that the Library of Congress uses to archive crucial media. The owner of Journalspace realizes that it's his own fault for not double-checking his IT guy's decisions, so I'm posting this in hopes that people might take steps to ensure that their backup procedures are secure to prevent something like this from happening. You can read more here. __________________ Gaybucks.com 100% exclusive American guys - hosted movie galleries - NATS - Boyfunk.com - Boysfeetclub.com - AJsCloset.com- SkylerDeVoss.com ICQ 272-995-402 Last edited by Gaybucks; 01-03-2009 at 06:22 AM..

01-03-2009, 06:25 AM	#3
Carmine Raguso So Fucking Banned Join Date: Dec 2008 Location: Prescott, AZ Posts: 2,158	Wow that really sucks. I don't know that i would just give up though with that many users. Maybe the users of the site would understand that shit happens? Seems like to many loyal users to just throw it away. Then again, what does a big greasy Italian from Brooklyn know...

01-03-2009, 06:27 AM	#4
Machete_ WINNING! Industry Role: Join Date: Oct 2002 Posts: 14,579	Most business use DLT/LTO backup tapes, or use a remote backup that archive the data on tapes each month. That said.. if the data was overwritten, it could still be restored by several methods. Both going after the SQL logs or simply recreating the overwritten date on the disk. Whoever wrote that story either dont know what they are talking about, or they are quoting people who dont know what they are talking about.

01-03-2009, 06:28 AM	#5
Machete_ WINNING! Industry Role: Join Date: Oct 2002 Posts: 14,579	PS. I would like to buy their 404 traffic

01-03-2009, 07:45 AM	#8
TisMe Confirmed User Join Date: Aug 2008 Posts: 1,719	CloneBox from bettercgi.com is what I use and suggest to others. Off site copies of sites ready to go live whenever needed or automatically if a server goes down.

01-03-2009, 07:54 AM	#10
Klen Industry Role: Join Date: Aug 2006 Location: Little Vienna Posts: 32,235	If you have multiple servers,then best is to set a cron on servers to do backup and cron on your local computer to download those backup so lets say if datacenter explode you will be prepared.

01-03-2009, 08:31 AM	#11
DraX Confirmed User Industry Role: Join Date: Oct 2002 Location: In wonderland.. Posts: 7,147	Very interesting story... I do make regular backups myself manually, so if there's anyone to blame it's me lol. But in this case with an estimate of 14k members i would keep fighting, shit happens!

01-03-2009, 09:04 AM	#13
Si Such Fun! Industry Role: Join Date: Feb 2008 Posts: 13,900	Oh dear what a way to go!

01-03-2009, 09:46 AM	#14
raymor Confirmed User Join Date: Oct 2002 Posts: 3,745	That really sucks for them. Thing is, we see this happen all the time. Everyone thinks it won't happen to them, or just assumes that their web host is taking care of it. Many, like journalspace, think RAID is backup of some sort. They learned what I've posted here several times - RAID IS NOT BACKUP. Here are some critical features of a proper backup system, as we learned while designing Clonebox (which is super backup on steroids): The backup must be on a different machine - spare copies on the same server can be handy, but they'll likely be lost at the same time as the main copy. Backups must be rotated, keeping several copies. Otherwise you end up like journalspace with two copies of crap when your backup runs right AFTER the files are deleted. Backups must be testable and tested - far too many people find out their backups are unreadable or otherwise unusable after it's too late. Backups must be automatic - while people may start out keeping up with a manual system, eventually they'll go months without a backup - right before disaster strikes. For web sites, backups must be immediately accessible - reloading 1 TB of content from tape or uploading it from home would take weeks, by which time most customers have canceled. Alphared, the hurricanes, and similar situations have taught us that backups must be off site - a backup provided by your host is useless when your host is raided by police, has their connection cutoff, has a fire, etc. Those are some things to check your current backup strategy against. If you're not confident in your current system, Clonebox will provide rotating backups for $5 / month, or four hot spare clones, which let you boot your backup and have the site live in seconds for $24 / month. __________________ For historical display only. This information is not current: support@bettercgi.com ICQ 7208627 Strongbox - The next generation in site security Throttlebox - The next generation in bandwidth control Clonebox - Backup and disaster recovery on steroids

01-03-2009, 09:58 AM	#15
qxm Confirmed User Join Date: Jul 2006 Location: NoHo Posts: 5,970	it is things like this that make my prostate feel enlarged.... dam I feel for those guys...just the thought of loosing so much work makes me wanna hurl __________________ ICQ: 266990876

01-03-2009, 10:00 AM	#16
StuartD Sofa King Band Join Date: Jul 2002 Location: Outside the box Posts: 29,903	no back up = stupid stupid stupid Raid it however you wish, you still need back ups and back ups of the back ups. __________________ This is me on facebook This is me on twitter

01-03-2009, 11:07 AM	#17
PhantomFrog Registered User Join Date: Jul 2006 Posts: 89	There are only two kinds of computer users---those who have lost data and those who will lose data. RAID ALONE IS NOT A BACKUP STRATEGY. It is a recipe for disaster. Print what Raymor posted (#15) about backups and put it behind a refrigerator magnet. Most businesses who post on GFY would be dead without their data. Data is a business asset ... just like money ... just like buildings ... just like traffic that converts like crazy. Gotta run. I need to run my rotational backups and take the backup disk to my safe deposit box before the bank closes. __________________ PhantomFrog.com....Premium Password Protection PhantomCart.com....CCBill + Phantom Cart = Clips Store Solution PhantomFlicks.com .Token-based Clips Store Last edited by PhantomFrog; 01-03-2009 at 11:10 AM..

01-03-2009, 11:12 AM	#18
Supz Arthur Flegenheimer Industry Role: Join Date: Jul 2006 Location: New York City Posts: 11,056	I do consulting for large companies and one of the I preach the most is backup / disaster recovery. If you look at statistics. Most companies who have an outage of 24 hours or more. Go out of business within a years time. If you have a site of that size, and you dont have a DR Site. With data replication. You are out of your mind.

01-03-2009, 01:58 PM	#19
skrinkladoo Confirmed User Join Date: Oct 2007 Location: South Florida - "Fuck Bitches and Fight Crime - It's all we do! Posts: 629	Raid will never replace backups. Raid 1 is at its best when considered a means for protection against drive failure. Cloud is considered a new form of a worry-free semi-cluster like solution - but it requires backup in most cases. 6 years, and 14,000 bloogers - they didnt learn this? Not a very solid business plan. __________________ * Andrew Love * Services: (1) Web Design, (2) Graphic Design, (3) E-commerce, (4) Database Development, (5) Web Analytics, (6) Targeted Lead Generation, (7) Marketing and Consulting Mainstream: Successful Website Landing Page Design \| ICQ: 360-83-9627

01-03-2009, 02:00 PM	#20
Miguel T ♦ Web Developer ♦ Industry Role: Join Date: May 2005 Location: Full-Stack Developer Posts: 12,468	Damn, that sure was a great way to start the new year. __________________ Full Stack Webdeveloper: HTML5/CSS3, jQuery, AJAX, ElevatedX, NATS, MechBunny, Wordpress

01-03-2009, 02:00 PM	#21
CrkMStanz Confirmed User Join Date: Jan 2008 Posts: 517	Offline backups... #1 important process for a server soooo underdone __________________ believe me - without free porn, just as many people will seek porn out on the Internet, and many more will pay if there is no free alternative, its not like sex is a fad - it can be milked much like any renewable resource - long term i wasn't born with enough middle fingers - Marilyn Manson

01-03-2009, 02:10 PM	#22
PR\|Jordan Confirmed User Join Date: Oct 2008 Posts: 244	This is VERY important - OC3/PR took it a step farther recently! We just upgraded our backup solutions and are migrating customers now! The new backup system is on a scalable redundant cluster capable of growing to 100 petabytes operating on R1soft (enterprise backup software www.r1soft.com). If a customers server dies for some reason - we can click 1 button and restore it on any hardware - after the restore is complete the server reboots and starts pinging again - our customers do not even need to reconfigure their OS or install their scripts - it backs up EVERYTHING. On top of this - it supports backing up MINUTELY! I have a customer who: Backs up every 5 minutes and stores the restore points for 24 hours (so 300+ restore points in a day) Backs up daily and saves restore points for a month Backs up Monthly and saves it for a year! Because they use incremental backups the space required for additional restore points is almost nothing. So - BACK UP! __________________ Jordan Jacobs - OC3 Networks / Pacific Rack - ICQ:1408 1732 - jjacobs[@]pacificrack.com These deals ROCK! Dual Core E2180, 2GB DDR2, 250GB SATA-2, 2000GB - $89 /month Quad Core Q9300, 2GB DDR2, 250GB SATA-2, 2000GB - $129 /month

01-03-2009, 02:17 PM	#24
RevTKS69 Confirmed User Join Date: Dec 2008 Location: West Texas Posts: 231	Chains of failure Something that tends to get lost in talking about data backup and recovery is the sequence of events that create the backup and events that can corrupt that process. As in the case of the RAID setup, it did what it was supposed to do...keep a copy of the data from drive1 on drive2. The problem here was two-fold, the data was lost/overwritten and THEN the second drive simply duplicated the missing/overwritten data. So, everyone starts talking about keeping copies on another machine, using off-site storage and other wonderful ideas. However, if the user came along right AFTER the data was overwritten and made a copy and stored it off-site they would still have a copy of nothing. Backup is not just about multiple copies, it is about ensuring data integrity throughout the process. Not only do you have to copy the data, you have to check and make sure that it is indeed the data you want and that it can be restored in a timely manner. There are many horror stories of IT departments that have a multiple/redundant copies of their critical data (including those wonderful tapes) and when disaster strikes they find out that all of their tapes, drive images, and other data backups are somehow unusable. __________________ www.sexyschiava.com - Amateur BDSM www.adultblogreview.com - Starting a new review site, positions available

01-03-2009, 02:45 PM	#27
Drake Hello world! Industry Role: Join Date: Mar 2003 Posts: 12,508	Terrible way to go down

01-03-2009, 02:52 PM	#28
hypedough Confirmed User Join Date: Sep 2007 Location: Now running NATS4: HypeDough.com! Posts: 3,743	Holy shit my balls would hurt for days after that. As I was reading it, whenever someone says mirrored I always worry. A mirror of nothing is going to be nothing, like in this case. Horrible situation all around. __________________ Ricky D :: Hype Dough President \| XBIZ.net \| ICQ 172-939-826 AIM+Skype HypeDough \| [NATS4] Kayden420: ['09 '10 '11 XBIZ Nominee \| Exclusive & HD] \| ThePornScout: [Exclusive + Reality \| Amateurs Want to Become Pornstars]

01-03-2009, 03:37 PM	#29
st0ned Confirmed User Industry Role: Join Date: Mar 2007 Location: Arizona Posts: 8,437	I recently found this out the hard way. One of my servers failed a little over a week ago and I lost 115 sites. I currently have the drive at a Data Recovery company but so far they are unable to retrieve anything. Was planning on starting 09 with a bang but this definitely set me way behind. I have been using Google cache to try and salvage what I can. Anyone know any other sites besides archive.org that may have a cache of my sites? That REALLY sucks for JournalSpace and all of the bloggers there. In their situation they should have been far more careful in regards to backing up their stuff since they had some 14,000 people relying on their service. I just run a bunch of free sites and assumed my host had backups. As always, I tend to have to learn my lessons first hand. __________________ Conversion Sharks - 1,000+ adult dating offers, traffic management, and consistently high payouts. We will guarantee and beat your current EPC to win your dating traffic! Skype: ConversionSharks \|\| Email: info /@/ conversionsharks.com

01-03-2009, 04:16 PM	#32
TyroneGoldberg Confirmed User Join Date: Sep 2007 Posts: 1,081	I don;t if a should laugh or feel sorry for them....it's funny, but not. they're like oh well, we'll just pack up and shut it the business down...the bloggers are crying...

01-03-2009, 04:35 PM	#33
Gaybucks Confirmed User Join Date: May 2004 Location: Sacramento, CA Posts: 451	One thing I didn't mention (and wasn't discussed in the article linked)... the site owner said in his blog that he immediately sent the discs to drivesavers, which is one of the better data recovery people, and they were the ones that determined the data could not be recovered. Given that it is (theoretically at least) possible to recover data even after the drive has been overwritten, I have to assume that either it was too expensive, or not practical, or Drivesavers wasn't equipped with that technology. All of the suggestions in this thread are good ones. I would trust Raymor's solutions... he's never steered us wrong... and for us, we rely on NatNet's backups but also keep most everything backed up locally on one of our RAID-5 arrays here... and the arrays themselves are backed up onto SDLT. __________________ Gaybucks.com 100% exclusive American guys - hosted movie galleries - NATS - Boyfunk.com - Boysfeetclub.com - AJsCloset.com- SkylerDeVoss.com ICQ 272-995-402

01-03-2009, 04:36 PM	#34
spacedog Yes that IS me. Bitch. Industry Role: Join Date: Nov 2001 Posts: 14,149	they could make mad loot with all that 404 traffic

01-03-2009, 05:20 PM	#36
JD Too lazy to set a custom title Industry Role: Join Date: Sep 2003 Posts: 22,651	oh shit... talk about a nightmare :/

01-03-2009, 05:32 PM	#37
Iron Fist Too lazy to set a custom title Join Date: Dec 2006 Posts: 23,400	Google Cache is my backup plan. __________________ i like waffles

01-03-2009, 05:42 PM	#38
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	I was going to suggest they could probably restore a good chunk of entries via archive.org (even if they just turned it into a historical archive), but they're following the "accept some, deny all" robots.txt model which means that archive.org isn't allowed to crawl the site. If that block has been in place for some time then there's no chance of any recent posts being in the archive. User-agent: * Disallow: / User-agent: kinja-imagebot Disallow: User-agent: kinjabot Disallow: User-agent: Mediapartners-Google* Disallow: User-agent: Googlebot Disallow: User-agent: msnbot Disallow: Hmm... Google, Google Adsense, MSN/Live, and kinja are the only bots permitted to crawl the site. Where's Yahoo? Blocked......

01-04-2009, 01:54 AM	#42
Jakez Confirmed User Industry Role: Join Date: Jan 2004 Location: oddfuturewolfgangkillthemall!!!!!!! Posts: 5,656	Been there, done that. __________________ [email protected] - jakezdumb - 573689400 Killuminati

01-04-2009, 04:04 PM	#44
papill0n Unregistered Abuser Industry Role: Join Date: Oct 2007 Posts: 15,547	holy shit wouldn't you be be pissed good advise in your post there gaybucks

01-04-2009, 04:10 PM	#45
Redrob Confirmed User Industry Role: Join Date: Oct 2004 Location: In a refrigerator box by the tracks. Posts: 4,790	The shit hit the fan on that one!!