Question about storing several TBs of images ... - GoFuckYourself.com

NALEM · 01-06-2016, 04:44 PM

We are working on a new project that will eventually host several TBs of images, no video.
If you have experience in managing such a large DB, I would greatly appreciate it if you would get in touch with me. We are looking for a bit of advise, to make sure we are going in the right direction with things.

Also we are looking for Server Admin (Linux) to work with us remotely.

Thanks in advance.

SilentKnight · 01-06-2016, 04:46 PM

Several TB's of just pics?

Whew...what are they - RAW format?

PuppetMaster · 01-06-2016, 04:46 PM

That's a lot of images.

NALEM · 01-06-2016, 04:53 PM

Quote:

Originally Posted by SilentKnight

Several TB's of just pics?

Whew...what are they - RAW format?

Nope

Quote:

Originally Posted by PuppetMaster

That's a lot of images.

Yes it is.

L-Pink · 01-06-2016, 05:04 PM

That's a lot of ......

Pac Man! · 01-06-2016, 05:05 PM

SBJ · 01-06-2016, 05:06 PM

I'd just store them on floppys and be done with it

j3rkules · 01-06-2016, 05:17 PM

Quote:

Originally Posted by Pac Man!

Please... no him again.

AdultKing · 01-06-2016, 05:26 PM

Quote:

Originally Posted by NALEM

We are working on a new project that will eventually host several TBs of images, no video.
If you have experience in managing such a large DB, I would greatly appreciate it if you would get in touch with me. We are looking for a bit of advise, to make sure we are going in the right direction with things.

You haven't really provided enough information to give you any useful advice.

However generally speaking if you are dealing with massive databases then using a standard dedicated server model is not the way to go.

You would be using some kind of object storage that was flexible so that it could size up or down according to your needs.

Amazon, OVH, IBM, Google, HP and countless other companies provide highly scalable and low latency storage solutions. The OVH Public Cloud is an interesting product to look at because it's lower cost than many competitors.

Then you need to engineer a way to store and retrieve your data objects from the data store.

Will you need a high level of redundancy? Will you need to index the data before returning it to the end user ? Do you need to scale up and down easily or are you going to provide a fixed retrieval solution on a dedicated machine ?

Engineering big data projects is non trivial and you really need to put in a great deal of work to design a system that will continue to meet your needs as you grow and as user demand scales up and down. You don't want to be paying for compute or storage that you're not using so something cloud based where you pay for what you use is desirable.

Some products to look at

https://aws.amazon.com/solutions/

IBM - Cloud Computing for Builders & Innovators

https://www.ovh.co.uk/cloud/

SilentKnight · 01-06-2016, 06:31 PM

Quote:

Originally Posted by SBJ

I'd just store them on floppys and be done with it

Yeah, go with the 5.25" dual-sided.

NALEM · 01-06-2016, 07:02 PM

Quote:

Originally Posted by SBJ

I'd just store them on floppys and be done with it

Will this be enough? This is all I have left.

NALEM · 01-06-2016, 07:13 PM

Quote:

Originally Posted by AdultKing

You haven't really provided enough information to give you any useful advice.

However generally speaking if you are dealing with massive databases then using a standard dedicated server model is not the way to go.

You would be using some kind of object storage that was flexible so that it could size up or down according to your needs.

Amazon, OVH, IBM, Google, HP and countless other companies provide highly scalable and low latency storage solutions. The OVH Public Cloud is an interesting product to look at because it's lower cost than many competitors.

Then you need to engineer a way to store and retrieve your data objects from the data store.

Will you need a high level of redundancy? Will you need to index the data before returning it to the end user ? Do you need to scale up and down easily or are you going to provide a fixed retrieval solution on a dedicated machine ?

Engineering big data projects is non trivial and you really need to put in a great deal of work to design a system that will continue to meet your needs as you grow and as user demand scales up and down. You don't want to be paying for compute or storage that you're not using so something cloud based where you pay for what you use is desirable.

Some products to look at

https://aws.amazon.com/solutions/

IBM - Cloud Computing for Builders & Innovators

https://www.ovh.co.uk/cloud/

"Design a system that will continue to meet your needs" - this is exactly what we all are paying close attention to now, well before launch. Use of a cloud based storage system combined with traditional servers is what has been the plan.

I have already forwarded your links for them to look into.

Thanks AdultKing.

AdultKing · 01-06-2016, 07:28 PM

Quote:

Originally Posted by NALEM

Use of a cloud based storage system combined with traditional servers is what has been the plan.

Depending upon the use case, traditional bare metal servers may be an expensive option, there are so many better alternatives now that you'd really want to justify using standard dedicated servers.

It really depends what you're building though.

NALEM · 01-12-2016, 11:13 AM

Quote:

Originally Posted by AdultKing

Depending upon the use case, traditional bare metal servers may be an expensive option, there are so many better alternatives now that you'd really want to justify using standard dedicated servers.

It really depends what you're building though.

Lead programmer sent me a message, that they have us set up on AWS.
Whatever he sees up there, he is loving it. He went from being stressed to
"this will work great". So a big thank you Adult King.

Ferus · 01-12-2016, 11:29 AM

One of our clients have +600TB images, with two Windows 2012R2 servers (on Hyper-V with 8Gig of Ram and 8vCPU). The Application frontend sees a SMB3 share that I can just add servers/LUN's to if needed. But so far it have a throughput of about 1,5mill IOPS.

The Storage is on a Tintri solution, but we are planning to set up a tiered storage for a nearline Archive/backup on the cheaper(slower) SAN.

It's really really simple to set up today, but it depends on what you want/need in regards to CDN or Clustering

MiamiBoyz · 01-12-2016, 12:13 PM

Everything goes away in the end...why bother

muthisdev · 01-13-2016, 11:35 PM

Quote:

Originally Posted by NALEM

We are working on a new project that will eventually host several TBs of images, no video.
If you have experience in managing such a large DB, I would greatly appreciate it if you would get in touch with me. We are looking for a bit of advise, to make sure we are going in the right direction with things.

Also we are looking for Server Admin (Linux) to work with us remotely.

Thanks in advance.

What kind of redundancy are you looking for?
Also, more important than how big the images are... How many images? 6 million rows isn't a huge database.

01-06-2016, 04:44 PM	#1
NALEM Confirmed User Industry Role: Join Date: Nov 2010 Location: Where ever Delta flies Posts: 3,134	Question about storing several TBs of images ... We are working on a new project that will eventually host several TBs of images, no video. If you have experience in managing such a large DB, I would greatly appreciate it if you would get in touch with me. We are looking for a bit of advise, to make sure we are going in the right direction with things. Also we are looking for Server Admin (Linux) to work with us remotely. Thanks in advance. __________________ "The time men spend in trying to impress others they could spend in doing the things by which others would be impressed."

01-06-2016, 04:46 PM	#3
PuppetMaster Confirmed User Industry Role: Join Date: May 2002 Location: The Corporate Boardroom Posts: 545	That's a lot of images. __________________ Recognize Old Skool GFY. Vote for people like AaronM, JFK, Juicy, baddog and others who pioneered this place!

01-06-2016, 05:05 PM	#6
Pac Man! So Fucking Banned Industry Role: Join Date: Dec 2015 Posts: 171	Nalem.com does dvtimes photo shoot

01-06-2016, 05:06 PM	#7
SBJ So Fucking Fabulous Industry Role: Join Date: Apr 2003 Location: Indiana Posts: 11,375	I'd just store them on floppys and be done with it __________________ Chaturbate - Make tons of money from Cams! *Name Silo - Domains under $9 and free ID protect*

01-06-2016, 04:46 PM	#2
SilentKnight Megan Fox's fluffer Industry Role: Join Date: Oct 2005 Location: shooting pool in Elysium Posts: 24,818	Several TB's of just pics? Whew...what are they - RAW format?

01-06-2016, 05:04 PM	#5
L-Pink working on my tan Industry Role: Join Date: Mar 2005 Location: Florida/Kentucky Posts: 39,152	That's a lot of ......

01-12-2016, 11:29 AM	#15
Ferus Bye - Left to do stuff Industry Role: Join Date: Feb 2013 Posts: 4,109	One of our clients have +600TB images, with two Windows 2012R2 servers (on Hyper-V with 8Gig of Ram and 8vCPU). The Application frontend sees a SMB3 share that I can just add servers/LUN's to if needed. But so far it have a throughput of about 1,5mill IOPS. The Storage is on a Tintri solution, but we are planning to set up a tiered storage for a nearline Archive/backup on the cheaper(slower) SAN. It's really really simple to set up today, but it depends on what you want/need in regards to CDN or Clustering

01-12-2016, 12:13 PM	#16
MiamiBoyz fgfdftre6 Industry Role: Join Date: Oct 2012 Location: In the closet with your dad! Posts: 6,690	Everything goes away in the end...why bother