Huge Project to edit in FCP X / How to optimal storage & Workflow
I am seeking for an advice on the optimal storage/editing workflow for a Huge project. I will try my best to explain the situation in detail as well as keep it short.
Me and my partner are freelancing filmmakers. We are doing a couple of projects here and there (every second month) that ranges from music videos, wedding videos, small corporate videos, small web commercials for catering companies etc…
I am currently editing on MacBook Pro 15’ 2016.
My cameras for the shoots are Panasonic Lumix GH4 / GH5 / Sony A7S / DJI Phantom 3 Pro.
I am editing via 4TB External Desktop HDD’s - USB 3.0 (Western Digital / Seagate)
I have only recently changed to FCPX - previously I was still editing the old school way on FCP7 - thanks film school - and I now find the workflow in FCPX so much easier, quicker and optimised for a quick turnaround that I often need. No more transcoding to Apple Pro res LT, FCP 7 workflow > Export / Compress / Transcode etc… I really like FCPX and I think I am going to stick with it. So far I have not encountered any problems in terms of performance on my MacBook while editing 4K natively, only once for one music video I was using MultiCam and I had to create Proxy for those couple of takes, otherwise I had a smooth editing experience editing with Original Media.
// THE PROBLEM //
We have a big project that I categorise as ‘separate’ from all our other film work, just for the reason of being so big. We are documenting a huge house being built from scratch - a project that we started filming 4 years ago - for the clients to have a short film / documentary 30-60 minutes of all the process and problems the building has encountered on its way (crane arrived late/ statues on the roof / golden fence being imported from Europe / welding of materials for window edges etc). The frequency of visits and our shoots are in average 1 visit/shoot every 3 months. Occasionally we might come twice or three times in one week if a lot of things/changes are being done tho the house, other times we haven’t been called for over 7 months.
I estimate that this project is going to continue for another 3-5 years until the house is finished. So far we treat all the footage shot as cataloging the progress as it still looks like a construction site so can’t even start editing the final product yet.
For this project I have decided to keep this on a separate storage HDD’s. So far it sits on 2x Desktop 4TB/Western Digital HDD’s and 1x 4TB Western Digital small HDD’s - in three separate locations. I CAN NOT AFFORD to loose any data for this project.
So far we have shot around 1.5TB of footage. I estimate that with the house being finished it will go beyond the 4TB current capacity.
// MY QUESTION //
// Sorry I am not very knowledgeable in the whole RAID set up systems. I have tried to read about it and watch youtube explanation videos but still not quite 100% get the benefits of the set ups.
Q1) How do I go about storing this project for another 5 years in the most effective / efficient way?
Q2) Once the house is finished - how do I go about editing this? Should I edit from the external HDD and have another external HDD - possibly a Thunderbolt as a ‘Final Cut Pro X library’ just solely for this project? How do I back up after each editing session I do to the other two HDD’s?
I have done some researching about the HDD hub that will clone and copy to the other two HDD’s in real time, should this be something I invest in now, or do this later? (in 5 years the technology might jump ahead again and the prices for a SSD’s might be way cheaper or other solutions might open up?
Q3) Is online storage such as Google Drive a possibility with editing such a huge project in FCPX?
The reason I am posting this now is to try and figure out if I can do any step at the moment to help my workflow in the future and for long-term. Or should I just store store store the footage for now and decide in the future?
Thank you for taking the time to read this and if you have any questions to make this more clear please don’t hesitate to ask.
Tadeas I'll chime in.
If you're storing on hard drives as it seems you are, I don't speculate that's too much of a problem. However if these drives stay dormant and powered off for a long time, I would spin them up at least once a year. There's technical docs about why it's good to do this, but I won't get into all of those details. I keep cloned bare 3.5 SATA archive drives and put reminders on my Calendar.app when to spin them up again, because I know I won't remember. Even if you only visit this project every so often in a handful of months you should be good though. Just be mindful of the connection(s) you're using with the hardware you use as those may change long term. If that becomes an issue, then adapters will do you just fine.
I tend to use RAID 10 mostly and that's likely due to the formats I've been editing with which relates to cost, performance, etc. I am a sticker for protection as well. Currently using two thunderbolt G-Speed Quad-Bay Studio units with 16 TB each (8TB). I cut narrative regularly which means lots of shots and multiple camera. Performance has been great with multicam editing for me. I've even gotten good playback from proxies on a LaCie portable rugged RAID with those same multicam streams. Ultimately you'll have to decide which RAID mode fits your particular situation.
I don't really see the need for a separate drive for your library and your media (in this case). I am sure others may contest that. FCP X is going to make regular and multiple backups in your user account Movies folder (unless you specify a different directory). The is means that you will have a backup of the library on another location other than the external drive; on your computer. When I am feeling extra cautious, I simply export XMLS from FCP X after a session. I regularly use a drive at home and one for work that have the same media, so between them I either update events from one to the to the other while having both libraries loaded in FCP X, or use an XML file. Though the XML file I use almost exclusively as an emergency backup and of course I export an XML of the final library.
I haven't tried online storage with FCP X. Can't speak to that, but your internet speeds need to be pretty robust if it's doable.
Keep on storing the shot footage and create new events. Back it up. Edit. Repeat.
As an aside, certain RAIDS are kind of an inherent backup anyway, but here's my archival process for backing up:
Wow, that sounds like a cumbersome project! ...from a we-can't-ever-get-started-editing standpoint anyway. 😁 Will be cool to be able to condense years of building into a half hour or hour film.
Q1: Sounds to me like you already have a pretty good system for accumulating and storing the footage. I really like the triple-location redundancy. If you start to go over your 4TB capacity, for storage/archival, I'd say just add another drive at each location. And like Tangier said, make sure to spin them up and do a read of the whole drive once a year or so.
Q2: I may be a bit different, but I like to organize my media in folders, usually starting w/ date, e.g. "2018.07.27 Panama City Beach Trip". Then for a new "project" (not an FCPX "Project"), I make a Library (or Event) to match that folder name, but I don't copy the media to the Library in most cases. So, you could possibly just create a Library on your MBP, hook up your working 4TB drive(s), import your media by linking not copying, and work on it that way. As long as you arranged your footage in the same hierarchy on each disk or set of disks, they would be interchangeable later if one were to fail, get lost, etc.
As Tangier said, FCPX will autosave Library snapshots as you go, so just make sure you are backing those up and/or copying offsite, and you'll be able to recover your work in the event of disaster. For better performance, prior to starting your edit, you may want to invest in a TB3 RAID and copy one of your source disk sets onto that to work from.
Q3: I can't imagine the latency would be tolerable in a purely online media workflow. However, I guess Google Drive would be caching files locally and syncing changes as you worked. Perhaps you could store your Library w/o media there for safe keeping? Not sure if that would demolish your Internet connection tho w/ all of the reads and writes that FCPX would be doing. And if you wanted to store your media in Google Drive, you'd obviously need a volume big enough to locally cache all that footage.
Hope this helps. Good luck!
Tadeas, you have some good questions here and your huge project sounds like a fun one. I've been working on documentaries for about 15 years so I understand huge. Currently I have about 90TB of media that I am accessing with much more archived on LTFS tapes in case we need it in the future.
Q1: The other guys have answered this one pretty well. I like cheep drives in multiple locations as long as you spin them up yearly. But I am guessing you are doing that after ever shoot when you copy more media to them. ☺
Q2: I wouldn't wait for everything to be finished before starting to work with the footage. I would start making selects after each shoot while it is still fresh in your head – favoriting clips, rejecting footage, keywording everything.
All of our footage is organized in the Finder by trip date and camera card. I always leave media in place when importing so my Library file is small and I can move it or back it up whenever needed. This is also essential when working with multiple editors. I can give a second editor a copy of the footage on a second hard drive, then we can pass edits back and forth via XML (or transfer libraries when custom or third party effects are used).
When offloading cards we use Hedge. It is great at making verified copies to multiple drives. I also use Goodsync to make sure two drives have the identical media (ie between editors). If I (or another editor) add footage to our local drive, I can sync the all the drives to match.
Q3: I have not had luck with storing massive amounts of media online. It takes too long to upload and costs to much to store the huge amounts of media we are dealing with.
Best of luck with your project!
Chad Greene : Mahoney Media : Minneapolis, Minnesota : 2x2.66 GHz 6-Core Intel Xeon, 12GB Ram, OS 10.11 FCPX, Kona-LHi
Wow thank you all for such great feedback.
> Chad / The only reason I am not editing yet is that I aslo agree with this quote:
"Don't think about this yet. Who knows what changes might occur in editing software in the next 3-5 years. Google or Apple might have exceptional cloud based AI software which could do 90% of the work for you in a fraction of the time. FCPX might change significantly, or you might be using a different editing program and starting from scratch."
I will check out Hedge but some other honourable mentions were: Carbon Copy Cloner or Chronosynch - any thoughts on those?
> Tangier / Fred : Would it be beneficial to have something like this and every time I do some backing up it would automatically back up to the other ones connected?
This also might relate to my other work hard drives – for example – when I want to edit a wedding video and at the end of the day or the project, I want to 'update' the other two HDDs with what I have done, wouldn't this be easier than copying one by one?
[Tomas Uher] "Carbon Copy Cloner or Chronosynch - any thoughts on those?
I have not used either of these programs so I can't comment personally, but I have seen people mention Carbon Copy Cloner.
As for your 1-3 Hard Drive cloner, I would suggest looking to see how it handles one changed file between two drives. I never want to recopy everything to a drive that already has 90% of the data on it. Maybe Carbon Copy Cloner can identify the differences between two drives and keep them equal? It would be nice to have one software for all... as long as it is easy to use and keeps the user errors to a minimum.
Chad Greene : Mahoney Media : Minneapolis, Minnesota : 2x2.66 GHz 6-Core Intel Xeon, 12GB Ram, OS 10.11 FCPX, Kona-LHi
Yep. That's similar to what I use. I've never used a 4-bay dock, but I have two two bay docks I got from OWC, one of them being thunderbolt connected and the other USB 3. Although I seldom ever have to use the second drive dock. Just make sure of the drive size limits for the dock. I have come across docks that don't recognize drives larger than 4TB. I tend to use WD Caviar Green for backups that go up to 6TB.