In Dire Need of Storage Solutions... can't keep editing like this!
So I've been editing for about two years and in this time my work has been interrupted by storage issues far, far too many times for it to be acceptable. Since starting I have been storing my media on external LaCie and Seagate drives, all FireWire 800, usually 2-3 TB each. Thing is, many times, at least 50% of the projects I work on, end up with the drive becoming unreadable halfway through my work process. Either the drives fail to mount suddenly, directories become unreadable, or just today a drive began spontaneously disconnecting every 5min, reconnecting, then disconnecting again. It's impossible to work this way. I've had to use DiskWarrior to save data, reformat drives, back-up on multiple drives in worry of repeat incidents, etc.
Can people help me with what is going wrong here? Surely someone with more experience has gone through similar situations? I work in the Mac OS X environment, typically using Premiere, Final Cut, and/or After Effects. I don't have any viruses or malicious software that could be responsible (these issues happen on different work stations; I have my own personal drives that never fail). Should I be using different forms of storage? I have long suspected that these drives are too slow for the constant work of rendering and playing back HD video but haven't found any proof to confirm my suspicions.
What do you use to store your media on? Should I get a RAID? How can I prevent these drives from constantly failing??? It's such a cumbersome burden that I can't imagine continuing to edit like this.
You're pain is one we've all felt - I have a stack of 11 USB drives that have failed over the last 3 years alone. Of course, I do use BRU and tape so I'm covered when the drives die. However, I also agree that the life expectancy of these things is far too short.
The issue seems to be related to the "cost" perspective in the eyes of the various USB / FW disk vendors - they're more worried about the price of the product rather than the expected life of the product. For comparison, an enterprise grade 2TB raw disk costs $220 in OEM quantities ($350 retail). And THAT'S just the drive. On the other hand, the consumer grade 2TB drives run under $100. You can see where the average USB or FW drive falls.
However, stepping away from consumer grade USB / FW drives really depends on your system. Do you have Thunderbolt or an available PCIe slot? If not, you're limited to a NAS-type solution. But, even those are either prone to failure (cheap) or very expensive. As with almost everything in the computer world, your choices are "High Quality", "High Performance", "Low Cost" - pick any TWO.
Finally, even if you do move to a RAID solution, you still face failures that can cost you data. While a RAID 5 solution can save you from a single drive event, it can't save you from a filesystem failure, accidental deletion, user mistake, or multi-drive failure in the array. This is where backup comes into the picture. Sure, copying your files to more than one array may be a short term solution, but only something that can be safely taken offline for extended periods will truly save you on those tight schedules when Mr. Murphy has his way. Both Blu-Ray and Tape are good options, but tape (LTO especially) offers a much higher capacity point and performance than any optical disk solution.
I know of at least 7 movies in the last 4 years that would have suffered incredible time and budget overruns (if they could have even been finished at all) if it had not been for LTO tape backup. In each case, the result of the failure and subsequent restore operation was the loss of less than a day of operation and creative time.
CTO - TOLIS Group, Inc.
BRU ... because it's the RESTORE that matters!
Thanks Tim, I've been looking through this forum and your tape suggestions which I'm looking into in more detail now.
I am running a 2011 iMac, so I do have a Thunderbolt option.
For my current short term solution, I am moving all my media from my external backup to my local drive and working off of that. Is this a good idea for the time being or will I run into another type of issue this way (I've never seen anyone work with their media off of their internal hard drive, so just curious).
The primary reason for not running your media off of your system drive is one of spindle access - if both the application AND your data are coming from the same drive, you'll see further slowdowns caused by the higher access requirements on that single drive. This is why all of the NLE vendors recommend an external data drive.
CTO - TOLIS Group, Inc.
BRU ... because it's the RESTORE that matters!
What is the computer you're using with these drives?
I have three LaCie D2 Quadras, an 8TB 4big and lots of WD Caviar/Scorpio blacks (and a couple 3TB greens I pulled out of "WD Elements" saving like $150) . I practically use bare drives like SDHC cards with a two-slot esata dock, even some drives in RAID-0 on the dock. I haven't really had any mentionable problems with any of these drives going between a Mac Pro with esata and a MacBook Pro with FireWire 800 (or handing drives off to clients) over the last couple years.
I use a three drive RAID-0 in my Mac Pro for a media drive, SSD for system drive and the 4Big for backup. The MBP has SSD for system and a Scorpio Black for a media drive in the optibay. Complete clean installs to a new drive keeping the old one on the shelf are done a few times a year. Three new media drives at least yearly, formatting and rotating the old ones into the three D2 Quadras. Then continuing to use the old drives from the D2's as bare drives in the dock or formatting for misc use (sometimes in RAID-0 and daisy chained via FW 800). Basically none of my "mission critical" drives are ever over a year old and starting with a clean slate of an empty RAID-0 just feels so nice. I may have lots of bare drives with backups of stuff I probably don't need but it is nice to have lots of new storage all the time and extra drives with backups that stick around longer than necessary. If I spend $1000 a year on storage and always feel reassured that I'm safe in the event of a failure, I think I made out pretty darn well.
For the constant clean installs I have one folder that travels to each new media drive that contains all my application installers and disc images, plug-ins, custom presets/settings/templates etc. and a note pad of all the license keys. I'm working on keeping my "Users" folder across installs but need to research whether its a good idea or even feasible. I don't bother with and end up carrying over messy drive structures, I keep the old one on the shelf and start fresh.
This system has kind of developed itself over the last couple years and works for me but a large portion of my work is archived "good enough" as a Blu-ray and a DVD cataloged and on the shelf and/or a YouTube upload and the file archived.
Hi Peter -
drives fail. Welcome to the club. If David Eaks has not experienced any drive failures yet, just wait - he will. Drive fail - end of story. When you look at brands like Lacie, G-Tech, Cal-Digit, etc. they all use the same internal hard drives from one of only a few companies - Hitachi (now owned by WD), Western Digital, and Seagate. Maxtor and Samsung drives are owned by Seagate. Lacie doesn't make drives, neither does G-Tech, neither does anyone else that you see advertise here on Creative Cow. I extensively use the same computers that David Eaks uses (Mac Pro computers) - which come standard with Western Digital Caviar Black internal hard drives, and guess what - they fail.
Archiving your data, and protecting your data is a fact of life in 2012 - just like paying taxes, or getting sick. Anyone tells you that "these drives will NEVER fail" is simply inexperienced.
Will having a RAID array help you - absolutely - modern arrays that do RAID 5 will allow you to have one failure without losing your data, and RAID 6 will allow you to have two drive failures without losing your data. But this does not mean that you won't get corrupt data. This is where programs like Disk Warrior come in, to save you. And free built in utilities like Apple Disk Utility Disk First Aid.
So, having a RAID is a good idea, but as Ricardo Reyes from TekRam said on one of these forums "RAID 5 is not archive", which is a great line. RAID's can have catostrophic failure, and so you must have secondary backups. These can be as simple as a single cheap SATA drive (like your Lacie, or even simplier like an OWC eSATA toaster with a bare drive) - all the way up to LTO tapes, like the ones sold by Tolis Group. LTO is VERY safe, but very slow compared to a hard drive. More and more people are moving towards LTO tapes for archive, as they experience catostrophic failure - once is enough for a lifetime, when you lose all your media.
SO, is all of this a nightmare, and is all of this giving you a headache - of course it does ! It gives everyone a headache. computers suck, and drives suck even more. The reality that film and videotape are truly going away now, and you only have your files, and if those files get lost or corrupt, you have no media, and nothing from your shoot - well, this is a very scary reality.
When will all of these problems go away - not any time soon. You will soon realize that the real expense of doing archives and backups is YOUR TIME to do this. It's something that you (and everyone else) doesn't want to do - but has to (or you get screwed).
So buy a RAID, by a Tolis LTO package, and be a little more broke, but sleep better at night.
Ditto on everything Bob says in his post about hard drives....but there are LTO solutions that perform well, thanks in part to LTFS formatting capability. Also, a good piece of software between the NLE and/or MAM that is metadata aware makes a very big difference when needing to restore your content - makes it so much easier to do project-based archive and restore rather than just "backing up" files and having to manually keep track of where things are. StorageDNA offers such a solution and at very reasonable prices - plus it can take some of the "suckiness" out of doing these tasks.
While Bob is correct in his comment about me not having suffered a devistating hard drive failure, I just wanted to add to my own comments.
Thanks to all the excellent information and advice here on the COW, I am prepared for the inevitable HDD failure. I dual record out on the field for redundancy, then back at the office my media is copied to three separate locations, my main editing RAID-0, the Lacie 4Big backup and to a LaCie D2 Quadra.
I realize my post may have come across like I'm carefree, willynilly with drives. But quite to the contrary, two of my LaCie D2s which I've labeled as "Runners" are generally used to move media from Mac Pro to MBP via sneaker net, essentially creating a fourth copy of my original media. I'll move the finished exports from a couple projects to the MBP to burn DVD/Blu-ray, keeping the Mac Pro open for editing.
I'm just lucky in my situation, being that 90% of my work is "done" after delivery and I will never use any part of it again in the future. I don't need real archive solutions, just safety until delivery. I keep the originals around for a few months for good measure.