A Somewhat Unique Situation
I have a gig editing, encoding, and managing educational content for an online company. In the past, I've handled this work by editing and storing the company's content on an internal RAID 0 that I backed up using firewire drives stored offsite. This is a terrible workflow, and the company has expanded to the point where the amount of footage being produced this year has more than tripled, so my 6 TB internal RAID 0 will no longer hold a year's worth of footage.
I am estimating that they will produce roughly 15 TB this year, including all associated project files. It's all standard def, usually delivered tapeless, and the editing is very rudimentary, a few edits per hour of footage. Most of the job is encoding, making DVD's, uploading, etc. Once more, the footage is only "good" for a year, so long term archiving beyond that is not important.
I'm looking for a better workflow. These files get shipped to me daily in 100-200 gig batches. 95% of the time, I edit, encode, upload .mp4's to a server, make DVD's, and I'm done with those particular files forever. Once in a while they might want to do some re-editing a month later, or an encode gets corrupted, and I need to re-output, but these are rare occasions.
At 15 TB, it seems like a high performance RAID solution holding all of those files would be especially expensive and unnecessary. I don't need constant access to all of these files at once, since 95% will never be revisited after DVD's have been burned and web files had been uploaded.
Are there lower cost "backup" RAID solutions that offer lower performance without sacrificing security ? Are they cheap enough that I could purchase two, using one as a backup that I move offsite?
Are there other backup solutions that could be paired with some sort of RAID?
Sorry if I'm rambling, but this job is an odd combination for me: extremely modest editing/performance requirements coupled with extremely high storage requirements.
I'd like to spend no more than $5000-$6000, but that may prove unrealistic.
Thanks for your input,
1. Ramping up your hardware to make double copies of the DVDs might be part of your next solution.
2. Your RAID 0 you are using now already gives you no protection from a big data crash. A bigger RAID 0 would lead to a bigger crash when the time comes. How about planning on a relatively small RAID 1 or 5 you can assemble yourself (buy the insides but use a dead computer`s case for the box).
3. This topic hosted by Bob Zelin`s name, and a lot of exchanges with John Heagy`s name, will net you plenty of answers awaiting pencil and paper. The exchanges though, are spread over almost a year`s time.
4. I don`t know how fast this backup works, but you might plan for and ask -here- if buying a few DAT24 scsi backup tape outfits, then buy "DAT24" (60 minutes = 8 GB, or 90 minutes = 12 GB) tapes from http://www.tapeonline.com, seen right above where you are reading or typing right now, is the backup portion`s answer. I haven`t seen any previous discussions of DAT24.
Ed, the concept you're describing is called tiered storage: online, nearline, and offline.
Online material lives on your high-performance RAID. It's immediately accessible. Nearline is a compromise between online and offline storage; it's a bit slower and not suitable for live production work, but still live. This might be a Drobo or an automated tape library. Offline storage is not immediately accessible. These could be manual tape libraries, DVDs or Blu-rays, or stacks of SATA drives sitting on a shelf.
[Ed McCarthy] "Are there lower cost "backup" RAID solutions that offer lower performance without sacrificing security ? Are they cheap enough that I could purchase two, using one as a backup that I move offsite?"
This might be Drobo. It's cheap, and it's way too slow for production use, but it's not bad for big, low-cost nearline storage.
Principal & Designer at Keen Live
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
RenderBreak Blog - What I'm thinking when my workstation's thinking
Creative Cow Forum Host: Live & Stage Events
Hey Ed, I'd agree with Walter -- we use the DroboPro for various backups. Not fast enough to edit from, but plenty fast to pull a few files and re-burn a DVD or re-render a master. You can get one loaded with 16TB of cheap drives for $2,000ish (B&H is running a special right now).
I think you'd be smart to build up "tiers" of storage, too. Have a nice RAID-5 drive for your fast editing, and a Drobo or something similar for slow (but safe) backups.