whats the best way to backup 200TB of data - amazon cold storage? or buy a tape backup - GoFuckYourself.com

necoeds · 11-10-2022, 02:49 AM

what I need to do for a backup of my studio hard drives with all my raw content from the past 22 years of filming
Currently I have 200TB locally here in Europe (on external drives)
I have duplicate external drives at my office in the USA, I have been syncing them using a vpn and a sync program for years.
BUT, this is becoming tedious, and i'm running into issues now with my RAW VR180 video files being like 250GB just one file, and my syncing software is fucking up
...so maybe you have some solution (I don't even know what to ask for specifically) ... I just want a cloud backup of my 200TB ... but this is COLD storage... I only need it, if one of my drives failshahere in Europe...
I probably should have said that first, I'm looking for a huge amount of cold storage - on tape is would assume... 2-3 months to recover a 16tb drive is ok for me, if something fails
I've searched and gotten a good idea of the cost of cold storage cloud storage - from amazon google etc... But I am hoping that just a tape system at mojo could be my solution
OR maybe just a tape system at my office in nebraska

k0nr4d · 11-10-2022, 05:47 AM

Wasabi is much cheaper then S3.

As for the cheapest way to archive data, and the safest - it's gonna be tapes...

Major (Tom) · 11-10-2022, 06:27 AM

Depending on your electrical grid reliability and budget, buy an old dell r610, load it with windows server, and an lsi raid card, several “rackable systems” sas enclosures, put in 80 or so wd nas red 4tb drives (possibly larger) and run it in a raid 60. It’s about 300 tb of redundancy. Get a good battery backup too.

Brad Mitchell · 11-10-2022, 08:10 AM

Check your email.. We have a new cold storage tier that is less expensive that Wasabi with more features so it's a better alternative to that and Amazon's glacier. We haven't put it up on the site yet or started marketing it... but I'm happy to put thousands of HDDs to work for ya

Brad

2MuchMark · 11-10-2022, 08:29 AM

Hi Necoeds,

What you want to do is balance your Recovery Time Objective (RTO), and Recovery Point Objective (RPO).

According to AWS at https://docs.aws.amazon.com/wellarch...bjectives.html, Your RTO would be defined as the maximum acceptable delay between the interruption of service and restoration of service. This determines what is considered an acceptable time window when service is unavailable.

Your RPO is the maximum acceptable amount of time since the last data recovery point. This determines what is considered an acceptable loss of data between the last recovery point and the interruption of service.

I recently attended a seminar by AWS and they have alot of backup solutions, not just S3. Even their cheaper solutions still give you fast access to your backups. I think their slowest one must be Tape though, and would expect much slower access to that.

I have been using AWS alot and am really happy with their prices, options and speed. I think Microsoft Azure and Google cloud may offer similar solutions but I have never used their backup / storage services.

Good luck!

plsureking · 11-10-2022, 09:00 AM

Quote:

Originally Posted by DukeSkywalker

Depending on your electrical grid reliability and budget, buy an old dell r610, load it with windows server, and an lsi raid card, several “rackable systems” sas enclosures, put in 80 or so wd nas red 4tb drives (possibly larger) and run it in a raid 60. It’s about 300 tb of redundancy. Get a good battery backup too.

yea keep it local. upfront cost but no monthly costs. if you choose one of these "cloud" solutions, you are married to them, paying monthly fees, and your data is at risk to any random issue.

you content and data only needs to be web accessible if you are serving it thru a website or node.

you can get 16TB drives on Amazon (prime not aws lol) for around $300.

#

DVTimes · 11-10-2022, 09:12 AM

Personally I would buy a few 2T external hard drives.

Back up your content to them.

While tapes may be cheaper, I suspect they may degrade in time.

If you plan on uploading to an external system, how long will it take you to upload that amount of data?

ravo · 11-10-2022, 09:36 AM

Quote:

Originally Posted by DVTimes

Personally I would buy a few 2T external hard drives.

A few??? Try 100+

Brad Mitchell · 11-10-2022, 09:49 AM

Certainly nothing could be cheaper long-term than just building some very large systems with raided 16TB drives - I could help you figure that out pretty easy. Having said that, if its the choice make sure you design a raid type with enough sparing. Also, any data that is just on a single system should have another copy somewhere else, for sure.

Brad

2MuchMark · 11-10-2022, 10:03 AM

Quote:

Originally Posted by plsureking

yea keep it local. upfront cost but no monthly costs. if you choose one of these "cloud" solutions, you are married to them, paying monthly fees, and your data is at risk to any random issue.

Not exactly. Your data is always yours and you always have access to it. A good cloud backup strategy would include backups in 2 different physical locations, at both an East Coast and a West Coast data center of AWS for example, which you can manage. So this means that even in the event of a catastrophic disaster, you can still access your data. And like any good datacenter, built-in redundancy, backups, hardware replacements, UPS's and especially security are all a part of the service.

Quote:

Originally Posted by plsureking

you can get 16TB drives on Amazon (prime not aws lol) for around $300.

#

This might be a good idea, but the question to ask is how valuable is the data. If the owner cannot ever lose the data and must have it accessible, then a cloud backup needs to be considered as a part of the overall backup strategy, since hard drives can fail, power failures can occur, OS's can crash, Ransomware attacks, etc etc.

plsureking · 11-11-2022, 01:56 AM

Quote:

Originally Posted by 2MuchMark

Not exactly. Your data is always yours and you always have access to it.

first of all, glad to hear you are still alive.

second, related to my first point. in case of death or bankruptcy, that data is no longer yours because you can't pay to host it anymore. at that point, you will need to make a local backup, which you can't afford because of the whole death/bankruptcy thing.

so even if you want to host it in the air (aka someone else's drives), make sure you have it locally backed up.

side note - most porn sites are already backed up on all the tubes

#

DVTimes · 11-11-2022, 02:42 AM

It is just to back up files, so no point having some fancy system designed for constant uploading data, as in vid editing.

As said, get a few hard rives, and back stuff up..

Remember hard drives can fail It is why I would not opt for huge storage, but more smaller drives, that way, if one fails, you loose less data.

I still would love to know how long it would take to upload all that data you have to something online, as you must have a fast connection. It is my big problem in the UK uploading 4k vids as it takes ages.

In a few years they may have 500TB drives that cost next to nothing, and all you need do is get your drives and copy everything to it.

Do not forget it is only a few years ago when 1gb SD cards costs over $100. Today they sell 250GB SD cards for next to nothing.

The advantage of buying hard drives, is that you can buy them as you need them too, to save cash. As it will take some time to fill them with data.

One thing too, is question if you do need RAW files, or can you convert them to MP4 or something? As that will reduce the file size. It may be worth having both RAW and say MP4 versions, in case of a crash.

zijlstravideo · 11-11-2022, 05:02 AM

What about the decentralized storage networks out there? They used to be a lot cheaper compared to the regular cloud storage options. Networks like Storj, Sia etc.

SpicyM · 11-11-2022, 06:48 AM

I would never rely on a third party service for backups, just like plsureking mentioned.

I would invest $4k, buy 10 drives, 20TB each and store them in a safe place. You can buy one for $390 on Amazon.

wankawonk · 11-11-2022, 07:34 PM

hard drives at home (even in RAID) are NOT suitable long-term storage devices

you'll get minimum 5 years before data loss, and realistically more like 10-20, but you will lose data within a couple decades. almost guaranteed.

Interestingly -- if you burn the data to write-only DVDs it will last at least 100 years on the DVDs. I believe this is the most robust long-term at-home data storage format.

that said....use amazon cold storage.

NoWhErE · 11-11-2022, 08:27 PM

How much would 200TB cost on Amazon Glacier (or S3)?

plsureking · 11-11-2022, 08:38 PM

Quote:

Originally Posted by wankawonk

hard drives at home (even in RAID) are NOT suitable long-term storage devices

you'll get minimum 5 years before data loss, and realistically more like 10-20, but you will lose data within a couple decades. almost guaranteed.

Interestingly -- if you burn the data to write-only DVDs it will last at least 100 years on the DVDs. I believe this is the most robust long-term at-home data storage format.

that said....use amazon cold storage.

sounds like a non-techie response. you only buy one computer or phone every 10-20 years? what happens when things break at your house? throw them in the backyard?

all equipment has to be maintained and updated. nothing lasts forever. the price of a home storage solution will be a fraction of any cloud or cold storage - and you own it.

#

plsureking · 11-11-2022, 08:44 PM

Quote:

Originally Posted by NoWhErE

How much would 200TB cost on Amazon Glacier (or S3)?

$737 USD / month

https://calculator.aws/#/addService/S3Glacier

#

Tubevideditor · 11-11-2022, 09:37 PM

Also looking for something similar.

Cloud storage is ok but without a fast internet connection, uploading tens or hundreds of terabytes won't be quick nor fun and you're tied into a monthly fee literally forever.
If anything goes down, you're without access to it.

Hard drives not really an option as they can fail over time. Will also need lots of them so will take up a lot of space.

LTO tape back up seems the best option for me but the tape recorder machines are surprisingly expensive for what they actually are but the actual tapes are cheap.
You also need an interface to use the player.

plsureking · 11-11-2022, 11:36 PM

education is your friend, and Youtube is your teacher.

.
.
but but but, hard drives fail over time!! ok...

or you can pay that $700-1000 a month and own nothing.

#

jamezon · 11-12-2022, 12:29 AM

lets say one of those 10 x 20 TB drives statistically fails within the next decade but 95% of the rest works fine for the next 20 years , will you survive that ?? or are you willing to pay 10times the price to archive the 99% level of security..then use tapes but deal with the issues (time, price, trickiness etc)..also please do avoid RAID, theres no need to have a system for offline storage where one disk depends on the other. also one thing, if you are in need to get back to some of the data, accessing the bare metal of the harddrive is much faster and easier then restoring it from tape. another thing with tapes, the bottleneck is the tape reading machine, they do make problems with outdated drivers or mechanical issues after 10- 15 years too sometimes,especially when urgently needed. a data center can cope with that but a single user.....

Major (Tom) · 11-12-2022, 02:39 AM

Quote:

Originally Posted by plsureking

yea keep it local. upfront cost but no monthly costs. if you choose one of these "cloud" solutions, you are married to them, paying monthly fees, and your data is at risk to any random issue.

you content and data only needs to be web accessible if you are serving it thru a website or node.

you can get 16TB drives on Amazon (prime not aws lol) for around $300.

#

The field narrows with which raid cards support 16tb drives

Major (Tom) · 11-12-2022, 02:41 AM

Quote:

Originally Posted by jamezon

lets say one of those 10 x 20 TB drives statistically fails within the next decade but 95% of the rest works fine for the next 20 years , will you survive that ?? or are you willing to pay 10times the price to archive the 99% level of security..then use tapes but deal with the issues (time, price, trickiness etc)..also please do avoid RAID, theres no need to have a system for offline storage where one disk depends on the other. also one thing, if you are in need to get back to some of the data, accessing the bare metal of the harddrive is much faster and easier then restoring it from tape. another thing with tapes, the bottleneck is the tape reading machine, they do make problems with outdated drivers or mechanical issues after 10- 15 years too sometimes,especially when urgently needed. a data center can cope with that but a single user.....

Raid 60 solves that. Each drive has parity

Major (Tom) · 11-12-2022, 02:43 AM

Quote:

Originally Posted by wankawonk

hard drives at home (even in RAID) are NOT suitable long-term storage devices

you'll get minimum 5 years before data loss, and realistically more like 10-20, but you will lose data within a couple decades. almost guaranteed.

Interestingly -- if you burn the data to write-only DVDs it will last at least 100 years on the DVDs. I believe this is the most robust long-term at-home data storage format.

that said....use amazon cold storage.

5 years? Going 7.5 and strong

plsureking · 11-12-2022, 06:54 AM

Quote:

Originally Posted by DukeSkywalker

Raid 60 solves that. Each drive has parity

ya some people can't do fucking research

1 drive fails in 20 years? you win!

keep a swap in the closet...

#

CurrentlySober · 11-12-2022, 08:16 AM

Quote:

Originally Posted by plsureking

[SIZE="5"]
or you can pay that $700-1000 a month and own nothing.

you forgot to say '& be happy?'

sandman! · 11-12-2022, 09:39 AM

Cheapest will be buying a bunch of 18+ tb hds and storing it yourself.

Online backups will cost more but then you have less work so depends on what you want

Freedom6995 · 11-12-2022, 10:26 AM

fwiw, I purchased a portable 16TB SSD drive on Amazon last year for around $80

plsureking · 11-12-2022, 05:57 PM

Quote:

Originally Posted by CurrentlySober

you forgot to say '& be happy?'

Quote:

Originally Posted by Freedom6995

fwiw, I purchased a portable 16TB SSD drive on Amazon last year for around $80

black friday coming up soon

#

Freedom6995 · 11-12-2022, 08:08 PM

Quote:

Originally Posted by plsureking

:
black friday coming up soon
#

I had to look, and it was indeed a Black Friday deal for $75. I would post it but the item is longer available.

money biz · 11-12-2022, 08:34 PM

Quote:

Originally Posted by wankawonk

hard drives at home (even in RAID) are NOT suitable long-term storage devices

you'll get minimum 5 years before data loss, and realistically more like 10-20, but you will lose data within a couple decades. almost guaranteed.

Interestingly -- if you burn the data to write-only DVDs it will last at least 100 years on the DVDs. I believe this is the most robust long-term at-home data storage format.

that said....use amazon cold storage.

I have cds from 2002 that have been in storage units in the heat outside that still work on old computer.

jsmih · 11-12-2022, 08:38 PM

Not big enough for your needs, but there are now 100 gb blu-ray archival disks called M disks).

NoWhErE · 11-12-2022, 09:11 PM

Quote:

Originally Posted by plsureking

$737 USD / month

https://calculator.aws/#/addService/S3Glacier

#

Yep. That’s a hard pass

plsureking · 11-13-2022, 12:27 AM

Quote:

Originally Posted by jsmih

Not big enough for your needs, but there are now 100 gb blu-ray archival disks called M disks).

CDs have 100 year life expectancy but the capacity is too low. their day came and went. i really enjoyed laser disc too.

hard drive capacity is growing every year. i upgrade my cms servers every 2 years just to max out capacity. my media/streaming servers after the upgrade last spring have 8x16TB drives. that's a buttload of porn!

#

jamezon · 11-13-2022, 04:41 AM

i have a dozen of cheap barracuda drives from 2005 that work to recover data from time to on the other side i have dozens of Dvds and Cds unable to read or constalty spitting out read errors. i think this thing with ever lasting CD and DVD is a Myth. there are more sensual to external factors like light, sun, dust , heat and prone to failure.

11-10-2022, 02:49 AM	#1
necoeds Confirmed User Industry Role: Join Date: Nov 2002 Location: Lincoln, Nebraska Posts: 2,453	whats the best way to backup 200TB of data - amazon cold storage? or buy a tape backup what I need to do for a backup of my studio hard drives with all my raw content from the past 22 years of filming Currently I have 200TB locally here in Europe (on external drives) I have duplicate external drives at my office in the USA, I have been syncing them using a vpn and a sync program for years. BUT, this is becoming tedious, and i'm running into issues now with my RAW VR180 video files being like 250GB just one file, and my syncing software is fucking up ...so maybe you have some solution (I don't even know what to ask for specifically) ... I just want a cloud backup of my 200TB ... but this is COLD storage... I only need it, if one of my drives failshahere in Europe... I probably should have said that first, I'm looking for a huge amount of cold storage - on tape is would assume... 2-3 months to recover a 16tb drive is ok for me, if something fails I've searched and gotten a good idea of the cost of cold storage cloud storage - from amazon google etc... But I am hoping that just a tape system at mojo could be my solution OR maybe just a tape system at my office in nebraska __________________ Message me to promote nebraskacoeds + network sites...

11-10-2022, 05:47 AM	#2
k0nr4d Confirmed User Industry Role: Join Date: Aug 2006 Location: Poland Posts: 9,228	Wasabi is much cheaper then S3. As for the cheapest way to archive data, and the safest - it's gonna be tapes... __________________ Mechanical Bunny Media Mechbunny Tube Script \| Mechbunny Webcam Aggregator Script \| Custom Web Development

11-10-2022, 06:27 AM	#3
Major (Tom) Anti Communist Industry Role: Join Date: Nov 2003 Location: Null Posts: 29,727	Depending on your electrical grid reliability and budget, buy an old dell r610, load it with windows server, and an lsi raid card, several “rackable systems” sas enclosures, put in 80 or so wd nas red 4tb drives (possibly larger) and run it in a raid 60. It’s about 300 tb of redundancy. Get a good battery backup too. __________________ My mother said, to get things done You'd better not mess with Major Tom

11-10-2022, 08:10 AM	#4
Brad Mitchell Confirmed User Industry Role: Join Date: Nov 2001 Location: Southfield, MI Posts: 9,810	Check your email.. We have a new cold storage tier that is less expensive that Wasabi with more features so it's a better alternative to that and Amazon's glacier. We haven't put it up on the site yet or started marketing it... but I'm happy to put thousands of HDDs to work for ya Brad __________________ President at MojoHost \| brad at mojohost dot com \| Skype MojoHostBrad 71 industry awards for hosting and professional excellence since 1999

11-10-2022, 08:29 AM	#5
2MuchMark Videochat Solutions Industry Role: Join Date: Aug 2004 Location: Canada Posts: 48,525	Hi Necoeds, What you want to do is balance your Recovery Time Objective (RTO), and Recovery Point Objective (RPO). According to AWS at https://docs.aws.amazon.com/wellarch...bjectives.html, Your RTO would be defined as the maximum acceptable delay between the interruption of service and restoration of service. This determines what is considered an acceptable time window when service is unavailable. Your RPO is the maximum acceptable amount of time since the last data recovery point. This determines what is considered an acceptable loss of data between the last recovery point and the interruption of service. I recently attended a seminar by AWS and they have alot of backup solutions, not just S3. Even their cheaper solutions still give you fast access to your backups. I think their slowest one must be Tape though, and would expect much slower access to that. I have been using AWS alot and am really happy with their prices, options and speed. I think Microsoft Azure and Google cloud may offer similar solutions but I have never used their backup / storage services. Good luck! __________________ VideoChat Solutions \| Custom Software \| IT Support https://www.2much.net \| https://www.lcntech.com

11-10-2022, 09:12 AM	#7
DVTimes xxx Industry Role: Join Date: Jun 2003 Location: UK Posts: 31,544	Personally I would buy a few 2T external hard drives. Back up your content to them. While tapes may be cheaper, I suspect they may degrade in time. If you plan on uploading to an external system, how long will it take you to upload that amount of data? __________________ The Affiliate Program

11-10-2022, 09:49 AM	#9
Brad Mitchell Confirmed User Industry Role: Join Date: Nov 2001 Location: Southfield, MI Posts: 9,810	Certainly nothing could be cheaper long-term than just building some very large systems with raided 16TB drives - I could help you figure that out pretty easy. Having said that, if its the choice make sure you design a raid type with enough sparing. Also, any data that is just on a single system should have another copy somewhere else, for sure. Brad __________________ President at MojoHost \| brad at mojohost dot com \| Skype MojoHostBrad 71 industry awards for hosting and professional excellence since 1999

11-11-2022, 02:42 AM	#12
DVTimes xxx Industry Role: Join Date: Jun 2003 Location: UK Posts: 31,544	It is just to back up files, so no point having some fancy system designed for constant uploading data, as in vid editing. As said, get a few hard rives, and back stuff up.. Remember hard drives can fail It is why I would not opt for huge storage, but more smaller drives, that way, if one fails, you loose less data. I still would love to know how long it would take to upload all that data you have to something online, as you must have a fast connection. It is my big problem in the UK uploading 4k vids as it takes ages. In a few years they may have 500TB drives that cost next to nothing, and all you need do is get your drives and copy everything to it. Do not forget it is only a few years ago when 1gb SD cards costs over $100. Today they sell 250GB SD cards for next to nothing. The advantage of buying hard drives, is that you can buy them as you need them too, to save cash. As it will take some time to fill them with data. One thing too, is question if you do need RAW files, or can you convert them to MP4 or something? As that will reduce the file size. It may be worth having both RAW and say MP4 versions, in case of a crash. __________________ The Affiliate Program

11-11-2022, 05:02 AM	#13
zijlstravideo Confirmed User Industry Role: Join Date: Sep 2013 Location: The Netherlands Posts: 805	What about the decentralized storage networks out there? They used to be a lot cheaper compared to the regular cloud storage options. Networks like Storj, Sia etc. __________________ Contact: email

11-11-2022, 06:48 AM	#14
SpicyM Confirmed User Industry Role: Join Date: Aug 2006 Posts: 4,534	I would never rely on a third party service for backups, just like plsureking mentioned. I would invest $4k, buy 10 drives, 20TB each and store them in a safe place. You can buy one for $390 on Amazon. __________________ no sig, sorry

11-11-2022, 07:34 PM	#15
wankawonk Confirmed User Industry Role: Join Date: Aug 2015 Posts: 1,017	hard drives at home (even in RAID) are NOT suitable long-term storage devices you'll get minimum 5 years before data loss, and realistically more like 10-20, but you will lose data within a couple decades. almost guaranteed. Interestingly -- if you burn the data to write-only DVDs it will last at least 100 years on the DVDs. I believe this is the most robust long-term at-home data storage format. that said....use amazon cold storage.

11-11-2022, 08:27 PM	#16
NoWhErE Too lazy to set a custom title Industry Role: Join Date: Sep 2005 Location: Canada Posts: 10,196	How much would 200TB cost on Amazon Glacier (or S3)? __________________ skype: lordofthecameltoe

11-11-2022, 09:37 PM	#19
Tubevideditor Confirmed User Industry Role: Join Date: Sep 2013 Posts: 336	Also looking for something similar. Cloud storage is ok but without a fast internet connection, uploading tens or hundreds of terabytes won't be quick nor fun and you're tied into a monthly fee literally forever. If anything goes down, you're without access to it. Hard drives not really an option as they can fail over time. Will also need lots of them so will take up a lot of space. LTO tape back up seems the best option for me but the tape recorder machines are surprisingly expensive for what they actually are but the actual tapes are cheap. You also need an interface to use the player.

11-11-2022, 11:36 PM	#20
plsureking bored Industry Role: Join Date: Aug 2003 Location: Metaverse Posts: 4,675	education is your friend, and Youtube is your teacher. . . but but but, hard drives fail over time!! ok... or you can pay that $700-1000 a month and own nothing. # __________________ #

11-12-2022, 12:29 AM	#21
jamezon Confirmed User Industry Role: Join Date: Apr 2019 Posts: 125	lets say one of those 10 x 20 TB drives statistically fails within the next decade but 95% of the rest works fine for the next 20 years , will you survive that ?? or are you willing to pay 10times the price to archive the 99% level of security..then use tapes but deal with the issues (time, price, trickiness etc)..also please do avoid RAID, theres no need to have a system for offline storage where one disk depends on the other. also one thing, if you are in need to get back to some of the data, accessing the bare metal of the harddrive is much faster and easier then restoring it from tape. another thing with tapes, the bottleneck is the tape reading machine, they do make problems with outdated drivers or mechanical issues after 10- 15 years too sometimes,especially when urgently needed. a data center can cope with that but a single user.....

11-12-2022, 09:39 AM	#27
sandman! Icq: 14420613 Industry Role: Join Date: Mar 2001 Location: chicago Posts: 15,432	Cheapest will be buying a bunch of 18+ tb hds and storing it yourself. Online backups will cost more but then you have less work so depends on what you want __________________ Need WebHosting ? Email me for some great deals [email protected]

11-12-2022, 10:26 AM	#28
Freedom6995 Friends of Venus founder Industry Role: Join Date: Jul 2010 Posts: 1,962	fwiw, I purchased a portable 16TB SSD drive on Amazon last year for around $80 __________________ Email: freedom6995 . protonmail.com

11-12-2022, 08:38 PM	#32
jsmih Confirmed User Industry Role: Join Date: Oct 2010 Posts: 334	Not big enough for your needs, but there are now 100 gb blu-ray archival disks called M disks).

11-13-2022, 04:41 AM	#35
jamezon Confirmed User Industry Role: Join Date: Apr 2019 Posts: 125	i have a dozen of cheap barracuda drives from 2005 that work to recover data from time to on the other side i have dozens of Dvds and Cds unable to read or constalty spitting out read errors. i think this thing with ever lasting CD and DVD is a Myth. there are more sensual to external factors like light, sun, dust , heat and prone to failure.