BUSINESS AND MARKETING: Business and Marketing Forum Business and Marketing Articles

Lots and lots of hard drives

COW Forums : Business & Marketing

<< PREVIOUS   •   FAQ   •   VIEW ALL   •   PRINT   •   NEXT >>
Rich RubaschLots and lots of hard drives
by on Aug 9, 2016 at 9:25:17 pm

I've got a client with a pile (hundreds literally) of external hard drives of every shape and size and form. They have camera footage from shoots from years ago, project files with ProRes converted footage and lots more. They want me to manage the drive collection. I was considering transferring everything to bare 3.5inch drives to consolidate and make access easier....we use bare drive enclosures that allow us to slide them in and out....works great.

But they are hard drives. I considered LTO but it might be too much of an investment and time, especially for the really old stuff that may never see the light of day.

Other way is to just have the drives out with the power supply and a bunch of cables and if we need something from a drive we just plug it in and hope for the best. It will be a mishmash of drives on the shelf for sure.

Anyone dealing with this and come up with a workable solution?

Rich Rubasch
Tilt Media Inc.
Video Production, Post, Studio Sound Stage
Founder/President/Editor/Designer/Animator
http://www.tiltmedia.com


Return to posts index

Tom SeftonRe: Lots and lots of hard drives
by on Aug 10, 2016 at 11:21:54 am

Could you figure out a fee for transferring each hard drive via Shotput Pro or similar to a RAID or LTO6 tape solution? This way, you get to keep the footage on a manageable system, and get paid for doing it.

We've got an LTO6 backup solution - works well after some initial teething problems. You can automate a transfer and leave it doing it overnight, with an MD5 and a pdf sent to you when completed. You can even bring multiple external drives into a project and copy them to one LTO tape. This way, you could then make duplicate copies of each drive, keep one yourself and send one back to your client (so it isn't your responsibility if someone spills coffee on it).

Co-owner at Pollen Studio
http://www.pollenstudio.co.uk


Return to posts index

Al BergsteinRe: Lots and lots of hard drives
by on Aug 10, 2016 at 1:49:54 pm

Depends on their budget. give them three choices. I would point out that those old drives will soon be unuseable (due to technology change) or dead. So keeping them around is minimalistic and suboptimal.

moving to large bare drives is another minimalistic step. no guarantee of footage being there if needed. next step is RAID which is where I'm at. better than bare but still needing backup, as theft , weather or fire can destroy them.

I bought 8TB RAID 1 for much of my legacy footage ( I've already lost some personal work I needed when a 2TB RAID 5 failed.) so it isn't foolproof. I backup to bare and offsite the bare drives to my bank's safe deposit box.

LTO would be best, obviously. good luck!

Al


Return to posts index


Mark SuszkoRe: Lots and lots of hard drives
by on Aug 10, 2016 at 2:38:46 pm

What about uploading each drive to rented cloud storage?


Return to posts index

Rich RubaschRe: Lots and lots of hard drives
by on Aug 10, 2016 at 3:40:07 pm

THANKS all. We looked at the cloud but with probably hundreds of terabytes the upload time alone would be prohibitive. LTO is a possibility but access and initial setup/copying is also time consuming.

I'm sure I'm not the only one in this situation considering how long we all have been dealing with hard media and not tapes. I'm dubbing a Betacam tape to DVD right now from the late 1990s and it still looks great and plays great. Wonder if that was on an old SCSI hard drive if it would have spun up!

More research, but in the meantime anyone else?

Rich Rubasch
Tilt Media Inc.
Video Production, Post, Studio Sound Stage
Founder/President/Editor/Designer/Animator
http://www.tiltmedia.com


Return to posts index

Ned MillerRe: Lots and lots of hard drives
by on Aug 11, 2016 at 2:18:13 am
Last Edited By Ned Miller on Aug 11, 2016 at 2:21:33 am

I think this is a question best asked on Creative Cow's post centric forums. There's 3 good ones where i've seen this question asked:

• FCPX
• Avid
• Premiere

There's real gurus there especially if you can ask Shane Ross who does mainly network TV shows so he handles mucho TBs.

Best of luck,

Ned

Ned Miller
Chicago Videographer
http://www.nedmiller.com


Return to posts index


Tom SeftonRe: Lots and lots of hard drives
by on Aug 11, 2016 at 12:40:56 pm

Look up YoYotta for the LTO tape option. It's a few hundred dollars but once you've spent a couple of hours getting used to it its very easy and efficient.

We are midway through backing up 170TB with it at the moment - it took maybe half a day of setup and learning time before we started backing up using a thunderbolt LTO tape deck. If your client is willing to spring for it, there are LTO decks that accept multiple tapes so a large spanned archive (more than 2.3TB raw or 6TB compressed) can be automated accross 3-4 tapes without you having to touch it in the 96hrs or so it takes to copy and check all data. The main issue with anything other than a raid is that you will need to charge your client to go back and check the data is still consistent after a couple of years (LTO tape can perish...) and this is time consuming over a multiple spanned archive. If you have one raid set to raid 10 giving maximum security, you could plug it in once a year, run a checksum overnight and that's it.

For cloud storage, I'm sure I saw Amazon were doing a nifty service in mainland USA for access to their glacier storage with a gigE raid that is posted to you and posted back to them once full (each one is over 24TB in capacity) for them to upload to glacier storage using their 10gE links.. The costs are then relatively low each month (a few cents per GB) meaning your client pays Amazon for an assured storage service but then have to pay a one off fee everytime you need to download the data. The only issue with this is that you don't have access to the rushes immediately, other than knowing what is stored where.

If you want to know more about the LTO Backups, drop me a line.

It all depends on how much your client is willing to pay for this service....

Co-owner at Pollen Studio
http://www.pollenstudio.co.uk


Return to posts index

Walter SoykaRe: Lots and lots of hard drives
by on Aug 12, 2016 at 2:15:18 am

LTO is the way to go. There are no other practical archival solutions. If your client really wants to hang on to this data, I honestly don't see what other choices you have.

LTO is a little expensive to get into, and it does take a lot of time (remember that one copy of a piece of data may as well be zero; we keep archives on two backup sets and store them in separate locations), but it's a lot cheaper than re-shooting. You can archive in off-hours, one drive at a time, and you can expect 15-30 years of shelf life from LTO.

Walter Soyka
Designer & Mad Scientist at Keen Live [link]
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
@keenlive [twitter]   |   RenderBreak [blog]   |   Profile [LinkedIn]


Return to posts index

Bob ZelinRe: Lots and lots of hard drives
by on Aug 17, 2016 at 12:46:58 pm

solutions from Imagine Products (it's PreRoll Post, not Shot Put Pro), and YoYotta have already been suggested.
Even if cloud sites worked, it would cost too much money to permanently store hundreds to terabytes on an Amazon S3 or Google Drive site (or similar - DropBox, iCloud, etc.).

So the alternative is a large RAID array, with a redundant backup. This way, it's your own "cloud site" - but it's not a cloud site - it's at their facility. You can readily purchase 16 bay and 24 bay RAID arrays, run it right on a Mac or PC, and put in 8TB 5400 RPM drives.
Sixteen 8TB drives is 128 TB, and after RAID 6 (to allow two drives to fail), it's 112 TB. 8 TB WD RED 5400 RPM drives are about $320 each. And you will have instant access to your media at any time - not having to spend days to restore from LTO. SO, what if your 128 TB chassis catches on fire - have a redundant array - so now you can have a second 128 TB array (you can daisy chain 8 chassis together, so you can have up to a Pedabyte of storage available) - and run a delta cloning program like ChronoSync or Carbon Copy Cloner (40 bucks), so when media gets added to the RAID array, it is automatically cloned to the redundant array, in case of disaster.

This will be the least expensive way to accomplish this. What everyone seems to forget, when archiving to LTO, is that it is VERY VERY time consuming to do this - and labor become the factor, above the cost of the hardware.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index


Rich RubaschRe: Lots and lots of hard drives
by on Aug 17, 2016 at 12:59:00 pm

Bob, once again, makes great points. Although hard drives are mechanical they provide the fastest throughput and cost effectiveness.

While I understand the Raid 6 concept, how would that be different than just having multiple backup hard drives? We could simply copy to 2 or 4tb drives and clone them....keep one set on site and one off site. Wouldn't that accomplish the same thing as a redundant raid setup? In that scenario I don't need to create a large RAID, but could just copy to simple drives with their clones.

And that is exactly what we are doing now. It has worked great for the past 5 or 6 years...knock wood I've only lost one drive but had a backup.

Lots to think about. Wonder what LTO 8 will bring in terms of cost, capacity and speed.

Rich Rubasch
Tilt Media Inc.
Video Production, Post, Studio Sound Stage
Founder/President/Editor/Designer/Animator
http://www.tiltmedia.com


Return to posts index

Bob ZelinRe: Lots and lots of hard drives
by on Aug 17, 2016 at 2:10:02 pm

when you have a single RAID array that is raid protected, you have ALL of your media instantly available at your disposal. And your single drives now become your archive backup. Instead of having shelves and shelves of drives, you have a single 3 rack unit box that has all your media. And if you are paranoid, you can have TWO RAID boxes, and be cloning from one to the other, so you always have TWO RAID 6 copies - PLUS the single drives that they came from which can now be off site.

As for LTO - yes its very safe. But as we have all seen, nothing lasts forever. How are you going to restore a DLT tape. How about an AIT tape. How about LTO1, LTO2, LTO3, and LTO4. How about non LTFS formats from companies like Cache-A or BRU ? Today, you are "safe" with LTO5 and LTO6, but LTO7 is the new thing - and don't worry - LTO8 will come out, and then LTO5 will seem like a total dinosaur, and people will say "how do I recover my LTO 5 tape.
The concept of "good for 30 years" is nonsense, when a SCSI 1 interface and drivers for your LTO are no longer available.

Data has to be constantly migrated. When you pay a company like Amazon or Apple to keep your data on their cloud site - they too are prone to obsolecense, but you are paying them monthly, so as their old Isilon drive chassis become obsolete, they too migrate the data to modern RAID arrays.

Trying to restore a DLT tape is like trying to play back a 1" or Quad Videotape. Good luck finding a working machine.
So what is the 20 year plan - THERE IS NO 20 YEAR PLAN - data must be migrated. Yup - film is the safest medium.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index

Rich RubaschRe: Lots and lots of hard drives
by on Aug 17, 2016 at 10:24:09 pm

Bob's right again. And by the way, I have a working Sony 1" machine!

Rich Rubasch
Tilt Media Inc.
Video Production, Post, Studio Sound Stage
Founder/President/Editor/Designer/Animator
http://www.tiltmedia.com


Return to posts index


Walter SoykaRe: Lots and lots of hard drives
by on Aug 19, 2016 at 3:10:51 pm

I agree with Bob that data migration is the problem with all archival, and that this gets laborious. I'd add that your data's importance should be reflected in your budget. For us, the value of our data drops a lot after a few years, so maintenance is not a big deal after a while.

As for storing old data on RAIDs, I believe in tiered storage. We have online storage (active projects), nearline storage (as many recent projects as we can fit), and offline storage (tape). Ultimately, I can add offline tape capacity a lot more practically than I can add nearline spinning rust capacity, but it does take labor to do it. (Though if I were less cheap, a robot would make this a lot easier.)

Personally, I find the idea of using the original hard drives as the off-site backup terrifying. I know some folks do regularly scheduled spin-ups of their "archival" hard drives in an effort to keep the lubricants from failing. Perhaps if you go this route, you'd consider something similar.

Walter Soyka
Designer & Mad Scientist at Keen Live [link]
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
@keenlive [twitter]   |   RenderBreak [blog]   |   Profile [LinkedIn]


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]