whats the best way to backup 200TB of data - amazon cold storage? or buy a tape backup
 

what I need to do for a backup of my studio hard drives with all my raw content from the past 22 years of filming
Currently I have 200TB locally here in Europe (on external drives)
I have duplicate external drives at my office in the USA, I have been syncing them using a vpn and a sync program for years.
BUT, this is becoming tedious, and i'm running into issues now with my RAW VR180 video files being like 250GB just one file, and my syncing software is fucking up
...so maybe you have some solution (I don't even know what to ask for specifically) ... I just want a cloud backup of my 200TB ... but this is COLD storage... I only need it, if one of my drives failshahere in Europe...
I probably should have said that first, I'm looking for a huge amount of cold storage - on tape is would assume... 2-3 months to recover a 16tb drive is ok for me, if something fails
I've searched and gotten a good idea of the cost of cold storage cloud storage - from amazon google etc... But I am hoping that just a tape system at mojo could be my solution
OR maybe just a tape system at my office in nebraska

Wasabi is much cheaper then S3.

As for the cheapest way to archive data, and the safest - it's gonna be tapes...

Depending on your electrical grid reliability and budget, buy an old dell r610, load it with windows server, and an lsi raid card, several “rackable systems” sas enclosures, put in 80 or so wd nas red 4tb drives (possibly larger) and run it in a raid 60. It’s about 300 tb of redundancy. Get a good battery backup too.

Check your email.. We have a new cold storage tier that is less expensive that Wasabi with more features so it's a better alternative to that and Amazon's glacier. We haven't put it up on the site yet or started marketing it... but I'm happy to put thousands of HDDs to work for ya :winkwink:

Brad

Hi Necoeds,

What you want to do is balance your Recovery Time Objective (RTO), and Recovery Point Objective (RPO).

According to AWS at https://docs.aws.amazon.com/wellarch...bjectives.html, Your RTO would be defined as the maximum acceptable delay between the interruption of service and restoration of service. This determines what is considered an acceptable time window when service is unavailable.

Your RPO is the maximum acceptable amount of time since the last data recovery point. This determines what is considered an acceptable loss of data between the last recovery point and the interruption of service.

I recently attended a seminar by AWS and they have alot of backup solutions, not just S3. Even their cheaper solutions still give you fast access to your backups. I think their slowest one must be Tape though, and would expect much slower access to that.

I have been using AWS alot and am really happy with their prices, options and speed. I think Microsoft Azure and Google cloud may offer similar solutions but I have never used their backup / storage services.

Good luck!

yea keep it local. upfront cost but no monthly costs. if you choose one of these "cloud" solutions, you are married to them, paying monthly fees, and your data is at risk to any random issue.

you content and data only needs to be web accessible if you are serving it thru a website or node.

you can get 16TB drives on Amazon (prime not aws lol) for around $300.

#

Personally I would buy a few 2T external hard drives.

Back up your content to them.

While tapes may be cheaper, I suspect they may degrade in time.

If you plan on uploading to an external system, how long will it take you to upload that amount of data?

A few??? Try 100+

Certainly nothing could be cheaper long-term than just building some very large systems with raided 16TB drives - I could help you figure that out pretty easy. Having said that, if its the choice make sure you design a raid type with enough sparing. Also, any data that is just on a single system should have another copy somewhere else, for sure.

Brad

Not exactly. Your data is always yours and you always have access to it. A good cloud backup strategy would include backups in 2 different physical locations, at both an East Coast and a West Coast data center of AWS for example, which you can manage. So this means that even in the event of a catastrophic disaster, you can still access your data. And like any good datacenter, built-in redundancy, backups, hardware replacements, UPS's and especially security are all a part of the service.


This might be a good idea, but the question to ask is how valuable is the data. If the owner cannot ever lose the data and must have it accessible, then a cloud backup needs to be considered as a part of the overall backup strategy, since hard drives can fail, power failures can occur, OS's can crash, Ransomware attacks, etc etc.

first of all, glad to hear you are still alive. :thumbsup

second, related to my first point. in case of death or bankruptcy, that data is no longer yours because you can't pay to host it anymore. at that point, you will need to make a local backup, which you can't afford because of the whole death/bankruptcy thing.

so even if you want to host it in the air (aka someone else's drives), make sure you have it locally backed up.

side note - most porn sites are already backed up on all the tubes :1orglaugh

#

It is just to back up files, so no point having some fancy system designed for constant uploading data, as in vid editing.

As said, get a few hard rives, and back stuff up..

Remember hard drives can fail It is why I would not opt for huge storage, but more smaller drives, that way, if one fails, you loose less data.

I still would love to know how long it would take to upload all that data you have to something online, as you must have a fast connection. It is my big problem in the UK uploading 4k vids as it takes ages.

In a few years they may have 500TB drives that cost next to nothing, and all you need do is get your drives and copy everything to it.

Do not forget it is only a few years ago when 1gb SD cards costs over $100. Today they sell 250GB SD cards for next to nothing.

The advantage of buying hard drives, is that you can buy them as you need them too, to save cash. As it will take some time to fill them with data.

One thing too, is question if you do need RAW files, or can you convert them to MP4 or something? As that will reduce the file size. It may be worth having both RAW and say MP4 versions, in case of a crash.

What about the decentralized storage networks out there? They used to be a lot cheaper compared to the regular cloud storage options. Networks like Storj, Sia etc.

I would never rely on a third party service for backups, just like plsureking mentioned.

I would invest $4k, buy 10 drives, 20TB each and store them in a safe place. You can buy one for $390 on Amazon.

hard drives at home (even in RAID) are NOT suitable long-term storage devices

you'll get minimum 5 years before data loss, and realistically more like 10-20, but you will lose data within a couple decades. almost guaranteed.

Interestingly -- if you burn the data to write-only DVDs it will last at least 100 years on the DVDs. I believe this is the most robust long-term at-home data storage format.

that said....use amazon cold storage.

How much would 200TB cost on Amazon Glacier (or S3)?

sounds like a non-techie response. you only buy one computer or phone every 10-20 years? what happens when things break at your house? throw them in the backyard?

all equipment has to be maintained and updated. nothing lasts forever. the price of a home storage solution will be a fraction of any cloud or cold storage - and you own it.

#

$737 USD / month

https://calculator.aws/#/addService/S3Glacier

#

Also looking for something similar.

Cloud storage is ok but without a fast internet connection, uploading tens or hundreds of terabytes won't be quick nor fun and you're tied into a monthly fee literally forever.
If anything goes down, you're without access to it.

Hard drives not really an option as they can fail over time. Will also need lots of them so will take up a lot of space.

LTO tape back up seems the best option for me but the tape recorder machines are surprisingly expensive for what they actually are but the actual tapes are cheap.
You also need an interface to use the player.

education is your friend, and Youtube is your teacher.



.
.
but but but, hard drives fail over time!! ok...



or you can pay that $700-1000 a month and own nothing.

:2 cents:

#

lets say one of those 10 x 20 TB drives statistically fails within the next decade but 95% of the rest works fine for the next 20 years , will you survive that ?? or are you willing to pay 10times the price to archive the 99% level of security..then use tapes but deal with the issues (time, price, trickiness etc)..also please do avoid RAID, theres no need to have a system for offline storage where one disk depends on the other. also one thing, if you are in need to get back to some of the data, accessing the bare metal of the harddrive is much faster and easier then restoring it from tape. another thing with tapes, the bottleneck is the tape reading machine, they do make problems with outdated drivers or mechanical issues after 10- 15 years too sometimes,especially when urgently needed. a data center can cope with that but a single user.....

The field narrows with which raid cards support 16tb drives

Raid 60 solves that. Each drive has parity

5 years? Going 7.5 and strong

ya some people can't do fucking research :1orglaugh

1 drive fails in 20 years? you win!

keep a swap in the closet...

#

you forgot to say '& be happy?'

Cheapest will be buying a bunch of 18+ tb hds and storing it yourself.

Online backups will cost more but then you have less work so depends on what you want

fwiw, I purchased a portable 16TB SSD drive on Amazon last year for around $80

:1orglaugh:1orglaugh

black friday coming up soon

#

I had to look, and it was indeed a Black Friday deal for $75. I would post it but the item is longer available.

I have cds from 2002 that have been in storage units in the heat outside that still work on old computer.

Not big enough for your needs, but there are now 100 gb blu-ray archival disks called M disks).

Yep. That’s a hard pass

CDs have 100 year life expectancy but the capacity is too low. their day came and went. i really enjoyed laser disc too.

hard drive capacity is growing every year. i upgrade my cms servers every 2 years just to max out capacity. my media/streaming servers after the upgrade last spring have 8x16TB drives. that's a buttload of porn!

#

i have a dozen of cheap barracuda drives from 2005 that work to recover data from time to on the other side i have dozens of Dvds and Cds unable to read or constalty spitting out read errors. i think this thing with ever lasting CD and DVD is a Myth. there are more sensual to external factors like light, sun, dust , heat and prone to failure.