Need advice on HDD configuration (a.k.a RAID 5: The Debate)
I just picked up a used 2010 Mac Pro 2.93Ghz 12-Core, 24GB memory, Radeon HD 5870 on eBay for $2275 and to my surprise (it wasn't in the listing) it has an Apple RAID card! :-D The current configuration is Bay 1 has a 1TB drive, and Bay's 2 - 4 have 2TB drives each for an additional 6TB (all original from Apple).
On my current 2008 Mac Pro that this is replacing, I have the following hard drive configuration:
Bay (1) -> 512GB SSD [os x+apps]
Bay (2) -> 2TB [stock media]
Bay (3+4) -> 2TB RAID 0 (2 x 1TB) [video projects]
Bay 1 is my boot drive with OS X and all of my applications. Bay 2 has my media drive where I keep royalty free music, stock video footage, digital juice files, etc. Bays 3 & 4 are a software RAID 0 (2 x 1TB) with all of my video projects so this is my editing drive. All of this is backed up to external drives and I have an external CineRAID 9TB RAID 5 that I keep archives on (Camera Archives and Project Archives).
So far on the 2012 Mac Pro I've put a 512GB SSD in the second DVD bay and put OS X + apps on that. This leaves me all 4 bays to configure for data drives. It's currently configured as a JBOD (1TB + 2TB+ 2TB+ 2TB) so I have no idea how the original owner had them configured or what they were thinking when they ordered it this way from Apple. I assume they may have used the 1TB as a boot drive and configured the 3 x 2TB as a RAID 5?
I'm trying to think of how to best leverage the Apple RAID card having never had one before. Should I use my existing pattern but make the RAID 0 a hardware RAID (obviously) and configure as I did before? I'm wondering if I should put the 3 x 2TB drives into a RAID 5? I could replace the 1TB with a 2TB drive and have 512SSD boot + 2TB media + 4TB RAID 5 (3 x 2TB).
I guess what I'm asking is how would you best leverage the Apple RAID card?
Is editing on a 3 disk hardware RAID 5 fast enough? (I guess I could format it and see for myself) As I said, I'm use to a RAID 0 and I know a RAID 5 won't be as fast as that but will it be "fast enough"? (whatever that is) I work in HD (no 4K) mostly with AVCHD AVC/H.264 formats.
I also work alone as an independent editor so there is no fibre channel/SAN or anything in my environment. This is a stand-alone Mac Pro in my home office.
What would you guys suggest? Pros? Cons? Strategies? Debates? ;-)
I would buy an additional small SSD and put it in the empty DVD bay. Now you have a total of FIVE drives. Put your OS on the SSD, then you can RAID all FOUR drives in the bay. Maximum raid throughput and storage. Maximum performance and speed off of the SSD for your OS and core programs.
[Mark Raudonis] "I would buy an additional small SSD and put it in the empty DVD bay. Now you have a total of FIVE drives. "Sorry if you missed that in my original post but that's the first thing that I did. :-D I have a 512GB SSD in the empty DVD bay with the 4 other bays filled so I have 5 drives.
I am considering getting a OWC PCI SSD. The advantage not only being an extra drive but you get SATA III 6GB/s throughput instead of SATA II 3GB/s throughput. They're expensive but they also have an eSATA port and I have an eSATA enclosure that I'd like to be able to use as well. I'm wondering if they are worth it performance wise? Perhaps that should be a new thread?
[Mark Raudonis] "... then you can RAID all FOUR drives in the bay. Maximum raid throughput and storage. "So, that's something I hadn't considered. I have always had a separate media drive that contains stock content (royalty free music, images, video, etc.) and a separate drive for editing. You're suggesting that I make just one big RAID 5 drive for media and projects. That kinda makes more sense because a 4 drive RAID 5 should perform better than a 3 drive RAID 5. Hmmm... That's something to consider. Thanks!
I have both a standard SSD and the PCI SSD, both from OWC on MY 2010 Mac Pro. The standard SSD gets around 250/200 (Read/Write) the PCI SSD gets 600/375 -- using the Black Magic speed test. I find the esata inputs on the PCI card helpful as well. I'm very happy with that purchase.
nothin' attached to nothin'
"Deciding the spine is the process of editing" F. Bieberkopf
[Herb Sevush] "I have both a standard SSD and the PCI SSD, both from OWC on MY 2010 Mac Pro. The standard SSD gets around 250/200 (Read/Write) the PCI SSD gets 600/375 -- using the Black Magic speed test."Herb, Thanks so much for that feedback. I've been using the Blackmagic Designs Speed Test as well. I've been configuring my RAID in different configurations and testing the speed of each. RAID 0 has been pretty linear. A non-RAID single 2TB disk is giving me around 121/118 Write/Read for a single disk, 228/243 for a 2 disk RAID 0 and 324/353 for a 3 disk RAID 0 (which is faster than my SSD which tops out around 257/270). A RAID 1 is giving me 123/177 which is the same write speed as a single disk and 1.5x read speed. I haven't built the RAID 5 yet but that's my next test.
It's great to see that the OWC is giving better results than the SSD which is constrained by the 3GB/s SATA II speed of the older Mac Pro motherboards. I seriously considering getting one now. Thanks again!
John, I don't have advice but I'm anxious to hear responses to your query because I have many of the same questions about what I SHOULD purchase.
I am also wondering about the best config for FCPX, given the library structure. In FCP7 we always kept the project file separate from the media drive. In X, it is very easy to fall into allowing everything to be stored in the Library, because of the ease of copying/transferring the whole shebang intact.
On your system, would you have the Library on the SSD or other single drive, with the "sync" alias pointers to media on your RAID? Or keep the entire Library on the RAID, where it could easily be archived to your other Cine RAID as a single transfer? Does it matter?
BTW in my own home office setup, I tend to prefer RAID 0 because I end up backing up media anyway, so RAID 5 becomes redundant. As I understand it, RAID 5 keeps you working after failure in a time-critical environment ... versus stopping to do a major restoration. Is that accurate?
[Douglas K. Dempsey] "In FCP7 we always kept the project file separate from the media drive. In X, it is very easy to fall into allowing everything to be stored in the Library, because of the ease of copying/transferring the whole shebang intact."It's a different way of thinking about projects. What's a project without the media? It's a useless metadata file really. A collection of edit decisions without the media does not a video make! So keeping projects with the media makes a lot of sense when you think about it.
It also depends on how you use Libraries! I use a new library for each "major" collection of projects. Definitely for each client. So, for example, if you are working on a TV show that has episodes, I would create a library for each Season where a normal season is 13 episodes. I would have a Season 1 library and then each Event would be a new episode. Each project is then that episode, or motion graphics used in an episode, or the trailer for the episode, etc. It make sense for me to have everything together like that. I can then backup all of my work by backing up the library and I know I have everything I need to move to another workstation and continue editing (like moving from my MacBook Pro to my Mac Pro) I do this with my Boris TV episodes today.
If, however, you are collaborating on a project where the media is on a shared drive and want your project to be separate on a local drive, then when you create a Library, you should select to leave the media in place. Then your library will only contain your project files with symbolic links to your media files which will be separate on another drive because you left them in place when you imported them. So with FCP X you can work both ways as you suggest. I just don't work that way.
[Douglas K. Dempsey] "On your system, would you have the Library on the SSD or other single drive, with the "sync" alias pointers to media on your RAID? Or keep the entire Library on the RAID, where it could easily be archived to your other Cine RAID as a single transfer? Does it matter?"I would keep the entire Library on the internal RAID and then still keep the external CineRAID for archiving finished products.
I just don't see the need to keep the project separate from the media. I need to be able to move between my MacBook Pro and Mac Pro and so I like having everything in the Library so that I only have to transfer the Library and know that I have everything. Once again, that assumes you use multiple Libraries and not keep all of your work in one big Library! My libraries are small enough that this makes sense for me.
[Douglas K. Dempsey] "BTW in my own home office setup, I tend to prefer RAID 0 because I end up backing up media anyway, so RAID 5 becomes redundant. As I understand it, RAID 5 keeps you working after failure in a time-critical environment ... versus stopping to do a major restoration. Is that accurate?"That's correct. A RAID 5 (or RAID 1 or RAID 1+0/10, etc.) is NOT a backup. It is purely for continuing to work after a failure. You still must backup any RAID system in case of catastrophic failure where a catastrophe would be loosing more than 1 drive in a RAID 5 or two drives in a RAID 10. Obviously losing a single drive in a RAID 0 would be a catastrophe as well.
As I said, I quickly fell into the habit of keeping everything in the Library on external RAID, and just designating the FCPX auto backup to my internal drive, so you theoretically have another copy of the metadata/project if everything goes south before proper RAID backup and you must recreate a media drive and relink everything.
I'll be interested to see what people say about maximizing effectiveness of your Raid card.
A raid 5 with 3 drives sucks up 33% of your drive space. I find even a 4 drive Raid 5 kinda sucks that it takes up 25%. 6 drives is sweet as you get full backup but it only eats up 16% of the space if my memory serves me.
[Bret Williams] "6 drives is sweet as you get full backup but it only eats up 16% of the space if my memory serves me."
A pendantic point to be sure, but I prefer to use the word "redundancy" instead of "backup" when discussing RAID arrays. RAID levels other than 0 give varying degrees of protection against disk failure, but no protection against the other sorts of failures that "backup" implies protection against (corruption, deletion, overwrite-tion, etc.).
Designer & Mad Scientist at Keen Live [link]
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
@keenlive [twitter] | RenderBreak [blog] | Profile [LinkedIn]
I would agree. Redundancy is a much better term. So yeah, redundancy of Library & AE/Motion projects & assets on the RAID, and auto backups of AE, Motion, FCP X on the internal, plus backup of the footage at least on a portable usb 3 drive if not elsewhere. When complete, it's all archived to a RAID 0.
"It's a different way of thinking about projects. What's a project without the media? It's a useless metadata file really. A collection of edit decisions without the media does not a video make! So keeping projects with the media makes a lot of sense when you think about it. "
When video was shot on tape, it made sense to keep the small project file on the system drive. The tape served as the archive and the project file could after reimporting the media recreate the edits/timeline if the digital media got corrupted or the media drive died.
Now u need to back up completely digital media.
I like to keep copies of my p2 cards in their own folders and then import them into FC assigned locations. Then the entire project has to be backed up on a different physical drive.
I wonder if I had the original card copies and just the project file if my projects could be brought back as they were. That would keep the back up archives smaller.
Mac Pro, macbook pro, Imacs (i7); Canon 5D Mark III/70D, Panasonic AG-HPX170/AG-HPX250P, Canon HV40, Sony Z7U/VX2000/PD170; FCP 6 certified; FCP X write professionally for a variety of media; teach video production in L.A.
I had an Apple RAID card at one point. It was an elegant solution, if not a bit finicky at times. But it never gave me the speed of and external RAID.
As for "is it fast enough", that's pretty subjective question... and I suspect that if you ask three people, you'll get four opinions. For myself, speed has many advantages, in terms of saving me time and saving my clients money. It also lets me experiment more. So for me, whether you ever do 4K or not, speed is always good. That's why I still use an external TB2 array even now on a screaming new MacPro with the largest SSD.
Insight Productions Corp.
"Criticism may not be agreeable, but it is necessary. It fulfills the same function as pain in the human body. It calls attention to an unhealthy state of things." - Winston Churchill
I used to to do Raid 5 and never had problems--but always a little worrisome if the drive went down.
What I do now is:
1. Keep a copy of original cam source media on an external bus powered drive--500gb, 1 tb. I have a library of a couple of dozen of them.
2. I transcode/optimize the media onto a large thunderbolt drive 8-12 tb.
3. I have a second, usual less expensive 8-12 tb drive that I clone from the main drive at the end of each sessions.
If my A drive fails, I can immediately go to my B-drive. If they both fail I'm screwed but then I still have the original camera media. Seemed like a good compromise between cost and peace of mind.
There is no one perfect solution. At least with this workflow, I have two good copies plus cam original. Raid 5, only had 1 good copy.
[Michael Hadley] "I used to to do Raid 5 and never had problems--but always a little worrisome if the drive went down."
I use a similar set. The array I use for daily editing is a RAID 0. The second large array I synch the editing drive onto is a RAID 5. Then I keep cards fcparch's and copies of the selected media on separate sparse external single drives, divided per year (fcparch's on one drive, imported media on another). If the main drive goes south I can clone back the RAID 5 without having to walk on thin ice during the process, still keeping on working, as I can still resist one more drive failure. The sparse drives I keep in a second location give me that extra security in case of a severe problem or things like theft or fire. 3Tb hard drives has dropped significantly, so I like the extra security, even if can look paranoid.
early 2008 MacPro, BM Intensity Pro, early 2008 iMac, 2014 MacBook Pro Retina, Blackmagic UltraStudio Mini Monitor, FCP7, FCPX, OSX 10.9.4
For turnkey stuff, we shoot and transfer to portable drive(s) on location. We don't reformat cards until we have to. Then copy camera footage to raid 5. The library also exists on the raid. Final cut makes backups automatically to the system drive. At any point the raid could lose 1 drive and be ok. Or if it went down completely then the project could be reconstituted from the library backup and the camera backup. When projects are done I mive them to a mirrored usb3 raid 0 which also holds stock media. When that is full I remove the drives and store each at a different physical location for theft/fire protection.
Just don't fall into the trap of assuming RAID5 is bullet proof. A few years ago we had an external (4 drive) RAID set at RAID5 and 2 drives failed. We lost everything. Whatever setup you end up with, in a professional setup it pays to keep additional backups of your raw footage and daily backups of your projects.
Yep. Raid 5 for in production stuff with project backup on internal with footage backed up elsewhere. My risk would probably be AE / Motion additions which are also in the raid 5. But those get backed up to internal periodically. Most elements for graphics are emailed or dropboxed, so that generally exists elsewhere. IOW, if something catastrophic happened, things could be put back together.