Backup solutions for big sites - GoFuckYourself.com

strobi · 04-20-2010, 11:34 AM

With more and more tubes popping up with terrabytes of data. How do these guys manage the backups?

RAID isn't an option, since drive failures/ drive copies always can go wrong.

Offsite backups are pain in the ass to move 10+TB every day at like 10 mbits...

Incremental backups could help, but copying the whole system from time to time is needed and runs in the two digits or more TBs.

How do the big hosting firms do those needed backups of clients with LOADS of needed disc space?

Anyway, discuss...

Barefootsies · 04-20-2010, 11:38 AM

Actually that's why I do RAID on all my boxes PLUS secondary back ups.

RAID should save you if your main hard drive fails. So your site is not down when you are doing a hardware swap out. Then you have a secondary system in place for your daily/weekly/monthly.

As for myself personally, I also have back ups in my office of websites and their content. So should all else fail, I could FedEx to host/datcenter/office and have them put it on their server at the office.

DateDoc · 04-20-2010, 11:39 AM

I'd think most of the bigger sites use a CDN which has all their content on multiple servers worldwide. For all those to fail seems very unlikely. That leaves a much smaller backup - db, etc.

strobi · 04-20-2010, 11:45 AM

Quote:

Originally Posted by DateDoc

I'd think most of the bigger sites use a CDN which has all their content on multiple servers worldwide. For all those to fail seems very unlikely. That leaves a much smaller backup - db, etc.

I guess this is one of the only options. What puzzles me how you can make a backup of a 20 TB site for example?

Just copying the files from A to B to do the backup will take a longgggggggggg time.

Cloud hosting would be a solotion, if the cloud is really robust.

Small sites you can always just make a tar/zip file and send it to another hoster/server. But doing this with 10TB+, madness!

strobi · 04-20-2010, 11:48 AM

Quote:

Originally Posted by Barefootsies

Actually that's why I do RAID on all my boxes PLUS secondary back ups.

RAID should save you if your main hard drive fails. So your site is not down when you are doing a hardware swap out. Then you have a secondary system in place for your daily/weekly/monthly.

As for myself personally, I also have back ups in my office of websites and their content. So should all else fail, I could FedEx to host/datcenter/office and have them put it on their server at the office.

Great backup plan you got there!

But doing RAID and daily/weekly, if you are sporting HUGE amounts of data?

How does that work?

Leaves me puzzled...

RAID with 20 drives would be doable, but backing that up :-(

DateDoc · 04-20-2010, 11:51 AM

Quote:

Originally Posted by strobi

I guess this is one of the only options. What puzzles me how you can make a backup of a 20 TB site for example?

Just copying the files from A to B to do the backup will take a longgggggggggg time.

Cloud hosting would be a solotion, if the cloud is really robust.

Small sites you can always just make a tar/zip file and send it to another hoster/server. But doing this with 10TB+, madness!

Imagine how long it takes to upload 20TB and populate the CDN

Barefootsies · 04-20-2010, 11:54 AM

Quote:

Originally Posted by strobi

Great backup plan you got there!

But doing RAID and daily/weekly, if you are sporting HUGE amounts of data?

How does that work?

Leaves me puzzled...

RAID with 20 drives would be doable, but backing that up :-(

Well, it sounds like you have more data then I do in that regard. But I do video heavy sites myself personally. So I feel your pain. Not to the 20TB degree, but I can relate.

Thanks on the kudos. Yeah, I try and make sure myself, and clients, are never going to be down. It costs more to do things this way with the RAID ARRAY. But it does make sure that if a hard drive fails, I do not need to worry about ordering a HD. I can do a simple swap, and tada. Client remains online while getting failed drive replaced.

KingRedOnion · 04-20-2010, 11:57 AM

Mirrored SAN with hourly, daily and monthly snapshots.

mikke · 04-20-2010, 12:00 PM

rsync + storage box in ovh.com, thats all

chupachups · 04-20-2010, 12:04 PM

2 online boxes with 2x750gb
50gb at one host
50gb at another host
1 online backup service on one of those boxes
1 backup server with backup service at home (keepit)
sending a usb memory to a friend every now and then
gmail
dvd and hard drive in safe
hidden 2.5 wireless NAS

Damn

strobi · 04-20-2010, 12:08 PM

Quote:

Originally Posted by mikke

rsync + storage box in ovh.com, thats all

Was looking into that too, they have super cheap 24 drive setups, but filling that thing over even a 1000 mbit line would take some time :-)

fpforum · 04-20-2010, 12:13 PM

RAID is great for one..Also, remote backups are highly recommended. Make sure you are keeping your backups not only on your main server, but on a remote server as well!

strobi · 04-20-2010, 12:17 PM

OVH is insane! 36x 2000GB SAS array, 10 gigabit uncapped unmetered incomming, for only 1K$/month. How do they make money on that? Lol!

ScriptWorkz · 04-20-2010, 12:21 PM

With something like rsync you would only be moving new data or data that's changed, not everything all the time. So there ya go.

Also i do offsite backups for some of my clients, so if anyone in here need a secure / specialized offsite backup solution at a reasonable price, hit me up and i'll see how i can help. sales [at] scriptworkz.com

04-20-2010, 11:34 AM	#1
strobi Confirmed User Join Date: Nov 2002 Location: Belgium Posts: 7,383	Backup solutions for big sites With more and more tubes popping up with terrabytes of data. How do these guys manage the backups? RAID isn't an option, since drive failures/ drive copies always can go wrong. Offsite backups are pain in the ass to move 10+TB every day at like 10 mbits... Incremental backups could help, but copying the whole system from time to time is needed and runs in the two digits or more TBs. How do the big hosting firms do those needed backups of clients with LOADS of needed disc space? Anyway, discuss...

04-20-2010, 11:38 AM	#2
Barefootsies Choice is an Illusion Industry Role: Join Date: Feb 2005 Location: Land of Obama Posts: 42,635	Actually that's why I do RAID on all my boxes PLUS secondary back ups. RAID should save you if your main hard drive fails. So your site is not down when you are doing a hardware swap out. Then you have a secondary system in place for your daily/weekly/monthly. As for myself personally, I also have back ups in my office of websites and their content. So should all else fail, I could FedEx to host/datcenter/office and have them put it on their server at the office. __________________ Should You Email Your Members? Link1 \| Link2 \| Link3 Enough Said. "Would you rather live like a king for a year or like a prince forever?" Last edited by Barefootsies; 04-20-2010 at 11:42 AM..

04-20-2010, 11:39 AM	#3
DateDoc Outside looking in. Industry Role: Join Date: Feb 2005 Location: To Hell You Ride Posts: 14,243	I'd think most of the bigger sites use a CDN which has all their content on multiple servers worldwide. For all those to fail seems very unlikely. That leaves a much smaller backup - db, etc. __________________

04-20-2010, 11:57 AM	#8
KingRedOnion Confirmed User Join Date: Feb 2007 Posts: 189	Mirrored SAN with hourly, daily and monthly snapshots. __________________ Trade Adult - 250+ TGPs looking for trades

04-20-2010, 12:00 PM	#9
mikke Confirmed User Industry Role: Join Date: Jan 2010 Location: Europe Posts: 1,327	rsync + storage box in ovh.com, thats all __________________ icq: 395 294 346 http://www.adultsubmitter.eu - submit any adult site to 20 directories from 1 form! now 20 domains! http://www.porndeals.eu http://www.ebonybangbros.com

04-20-2010, 12:04 PM	#10
chupachups Confirmed User Join Date: Dec 2002 Location: Sweden/Spain you sum bitch! Posts: 6,576	2 online boxes with 2x750gb 50gb at one host 50gb at another host 1 online backup service on one of those boxes 1 backup server with backup service at home (keepit) sending a usb memory to a friend every now and then gmail dvd and hard drive in safe hidden 2.5 wireless NAS Damn __________________

04-20-2010, 12:13 PM	#12
fpforum Confirmed User Industry Role: Join Date: Dec 2006 Location: USA Posts: 1,045	RAID is great for one..Also, remote backups are highly recommended. Make sure you are keeping your backups not only on your main server, but on a remote server as well! __________________ Email: strikedata \|@\| gmail.com Skype: mistercashman

04-20-2010, 12:17 PM	#13
strobi Confirmed User Join Date: Nov 2002 Location: Belgium Posts: 7,383	OVH is insane! 36x 2000GB SAS array, 10 gigabit uncapped unmetered incomming, for only 1K$/month. How do they make money on that? Lol!

04-20-2010, 12:21 PM	#14
ScriptWorkz Confirmed User Industry Role: Join Date: Jul 2007 Location: Intraweb Posts: 274	With something like rsync you would only be moving new data or data that's changed, not everything all the time. So there ya go. Also i do offsite backups for some of my clients, so if anyone in here need a secure / specialized offsite backup solution at a reasonable price, hit me up and i'll see how i can help. sales [at] scriptworkz.com Last edited by ScriptWorkz; 04-20-2010 at 12:23 PM..