Creative Studio Array / Raid / HD Headache
So I would love to have some input on how some of you guys handle your data storage. We are currently operating on 45TBs of data and I have a nightmare of management of these data pieces over a few pieces of RAIDS and HDs.
Here is our current set up.
- (1) - OWC QX2 16TB in RAID 5
- (2) - OWC MERCURY RACK PRO 16TB in RAID 5
We used to also have a MAC Pro until it died last year with 12TB of full capacity data, however since the processor died we moved everything over to the RAIDs.
We are about to be filled out of our current stack of drives and I want to go on a binge of backups. For back ups I have been ordering bare internal drives and storing them in hard shock proof cases.
My concern with this is that we only plug these drives in 3 times a year for something we need on backup, etc. We use something like this to access the data on the bare drives:
So looking forward... I had a IT guy who has dealt with video production before and one of his other firms used the:
Then also had a backup at an off-site location. This is great, however whenever we stream 2-3 computers of video over the server (gigabit-ethernet) its a bit sluggish. Since we had that problem really bad even with our current set up we moved all of our transcoded footage to XDCAM instead of PRORES (We shoot 5D Mark III and transcode). This helped a great bit and the resolution looks great. It just enables us to stream a few computers of data much better than pro-res.
But I am looking for the next thing. Something that will take us for a few years and really make the process sing.
-I once worked at a video production agency that had 4 bay drives that you could eject and they had a board of HDs they could access for backup and work on old content. What is this called? Is this "Near Line" storage? Maybe buying something like this G-Tech and setting up the drives as individual RAID and eject them on need? Then have a separate backup drive for the content.
-I could also get this thing, but I am not sure this is going to be the solution to my problems.
-Lastly I am just contemplating getting another QX2 from OWC but then I am still stuck in my ways.
What do you guys do? Any suggestions for this massive mess of a storage problem?
Hi Ethan -
here is the quick answer - then I will go into detail.
SPEND SOME MONEY. No, I said this wrong - SPEND A LOT OF MONEY.
There are countless manufacturers, that you see advertise right here on Creative Cow - but let's not get bogged down with little detail like Creative Cow's sponsors - you have been buying from OWC - so let's talk OWC - buy the OWC Jupiter series of PROFESSIONAL drive arrays, that are huge professional RAID protected drive arrays, that can be daisy chained, that can hold all your data safely.
You mentioned Synology - this is one of the lower end NAS systems that can hold your data. With a 1G Ethernet interface, this can be a "near line" storage system for you - which means it's a big drive array, that is accessable to all your editors on the network, but is kind of slow. Now companies like Synology, QNAP, Netgear, and Drobo all sell similar NAS solutions that are perfectly fine backup solutions, and can work as near line storage, but if you want something FAST ENOUGH to handle your 2 - 3 workstations, you need 10Gig Ethernet interfaces. And guess what - this costs MORE MONEY, and companies like Synology (and others) all sell this stuff (just like OWC sells the Jupiter line of storage), but this costs MORE MONEY, so your cheap ass boss is going to have to get a crow bar and open up his wallet, and buy the RIGHT EQUIPMENT to make all of this work for you and your company.
There are LOTS of solutions out there, that do PROFESSIONAL archiving (LTO tape solutions), and near line storage (Synology, QNAP, Netgear, Thecus, Drobo), and PRO solutions that do TRUE SHARED STORAGE (like countless forums on Creative Cow, that are discussed here over and over, and have dedicated forums) like Small Tree, Studio Network Solutions, Facilis, EditShare, GB Labs, Apace Systems, Maxx Digital, and yes - OTHER WORLD COMPUTING - but THEY COST MONEY - so the bottom line here is that you can have exactly what you want, but they aint' gonna cost $499. You can have EXACTLY WHAT YOU WANT but you have to be willing to pay for them.
OK, lets get into details -
you want Synology -
look ! a professional NAS system with 10GbE interfaces - and guess what, Synology has lots of these
you want OWC ?
you don't get much better than this - but this don't cost no $499.
while G-Tech no longer sells large drive arrays for professional shared storage enviornments - this is a teriffic product -
while this similar G-Tech product is a piece of crap -
and the list goes on and on, for EVERY manufacturer (Cal Digit, Sonnet, Maxx Digital, bla blah blah). They all have professional gear that works great, and they all sell crap. You want crap, you will pay a cheap price. You want good - you will pay for it.
So anyway - you can buy the big fancy drive array, and now you will have a mess of horrible data on your wonderful, reliable RAID protected big 64 TB drive array - and it's still a mess. How do you organize this - ASSET MANAGEMENT - from companies like SquareBox Systems CatDV, or AxleVideo (or others). And some shared storage systems, like Studio Network Solutions offer free easy to use shared storage products like "Share Browser" with their products.
There are lots of solutions out there Ethan, but you unfortunately are not going to find these in the bargain basement bin at your local mail order on line store.
Rescue 1, Inc.
What Bob said plus let me emphasize BACKUP TO TAPE.
All of folks you that continue to use drives for long term storage are asking for trouble. We are loosing movies and other creative output because folks are not backing up/archiving to tape.
Did I mention you should BACKUP TO TAPE. As Bob mention that currently would be LTO6. And not LTFS IMHO.
Believe it or not, the best long term storage, particularly for "finished" movies is film. Kodak has products specifically designed for long term archiving.
Good luck with your upgrading,
Robert A. Ober
A big part of why film is "best" is because it's essentially self-describing by lacking an obscured encoding. The digital content involves multiple layers of encodings, and even if all of them were openly documented (which many aren't) it's that layering that turns unwinding it all in e.g. 20 years a really serious problem. But conversely film isn't the best in that it isn't original content, it's the baked in totally rendered final output of a process. It's a deliverable, even if it's merely put into a freezer.
LTFS is supported on LTO6, they aren't mutually exclusive. It's well recognized that it only addresses part of the problem. There's some interesting, not yet completed, work by SMPTE progressing on AXF. This attempt recognizes the problem isn't just about media, let alone one kind of media. While LTFS was openly documented, and somewhat self-describing, it doesn't really sufficiently solve most of the pressing archiving concerns. Hence why this work is being done by SMPTE rather than the storage industry (let alone a narrow slice of it).
[Chris Murphy] "LTFS is supported on LTO6, they aren't mutually exclusive."
Wasn't say they were. My point was to suggest that folks not use LTFS due to some of it's issues. It might be OK in a larger automated environment but for the smaller and one man shops using something like BRU (I am not associated with them other than I like and use the product.) would probably work better and you can make a CD with what is needed to restore the tape.
Glad you chimed in, Chris.
Y'all be cool,
Robert A. Ober