Project file size - how much can FCPX handle - transcoding options
I’ve got the following scenario: I’ve got footage (5D, C100, FS7) that adds up to 195,92 GB. Now I’ve imported it all into FCPX with the Transcoding options „Create optimized media“ and „Create proxy media“. First question: Does it even make SENSE to use both or if I use Proxies I should untick create optimized media?
Now my FCPX project ends up being 571,23 GB with all the footage imported. I’m quite shocked that 195,92 blows up to this size but I guess it’s simply math because I create 2 different files: Proxies and Optimized so of course if we double 195,92 GB and add 100 more the file size makes sense.
Still I worry: How big can a .fcpbundle file be and how much can FCPX handle. OF COURSE I know it depends on what machine you run and what your machine can handle. My fear is this: For a future project I will probably end up with 40 TB of FOOTAGE alone. If I transcode it using Proxies AND Optimized media the FCPBUNDLE will end up being MASSIVE. Can FCPX handle something like this or will it be an overkill and what are the minimum specs to handle this?
I’m on a 2012 iMac with a 3,4 GH Intel Core i7, 32 GB RAM and an NVIDIA GeForce GTX 680 MX. I fear this machine will struggle a lot under such a heavy load.
The bundle is just a folder so don't worry too much about its size. However at 40TB any system is going to be taxing its limits especially on the capabilities of the storage solution.
FCPWORKS - FCPX Workflow
FCP Exchange - FCPX Workshops
XinTwo - FCPX Training
[Michael Paul] "...footage (5D, C100, FS7...Now my FCPX project ends up being 571,23 GB with all the footage imported. I’m quite shocked that 195,92 blows up to this size but I guess it’s simply math because I create 2 different files: Proxies and Optimized...How big can a .fcpbundle file be and how much can FCPX handle....For a future project I will probably end up with 40 TB of FOOTAGE alone. If I transcode it using Proxies AND Optimized media the FCPBUNDLE will end up being MASSIVE. Can FCPX handle something like this or will it be an overkill and what are the minimum specs to handle this?...I’m on a 2012 iMac with a 3,4 GH Intel Core i7, 32 GB RAM and an NVIDIA GeForce GTX 680 MX. I fear this machine will struggle a lot under such a heavy load."
It didn't blow up to 571GB because of both proxy and optimized but because your original content is H264 and full-res optimized is much lower compression. Optimized media will often be several times the original H264 size.
40 TB of footage is a huge amount and typically would require a workstation-class machine like a top-spec Mac Pro with a very large Thunderbolt array. Then multiply that by at least 2x since it must be backed up. Even then you can't blindly transcode all the content. On a higher-end machine you typically don't need to transcode H264 1080p for performance reasons, and certainly not when skimming the content in the Event Browser. Once you've marked the initial favorites and whittled down the data to a few % of the original size, you might want to transcode that.
I don't know whether FCPX can handle 40 TB of footage. If split across multiple libraries it might, given enough horsepower. One problem is at very large event sizes, FCPX does a lot of 4k and 8k random I/Os to build and maintain the thumbnails in the Event Browser. Even though a large spinning RAID array has good sequential performance, it doesn't help on small random I/Os. An SSD array would help but those are very expensive at the 40TB size. However OWC makes one: http://blog.macsales.com/38713-owc-announces-40tb-ssd-thunderbay-4-mini-rai...
Re the largest single library, it would be interesting if someone did some scalability studies on how factors like multiple events, workflow and various hardware configurations shape the achievable upper limit. However I don't recall seeing anything like that.
FCPX is an integrated media database and editor. Managing a 40TB image database is a difficult task, regardless of actually editing. It's possible a "lesser" editor without the database could handle that but how would you organize and find the clip ranges to edit? That in turn pushes you toward a stand alone Digital Asset Manager, which simply kicks down the road the database task. Due diligence would require you do scalability and stress tests on that DAM before committing production data to it. That in turn would lead to the same questions you're asking now -- hardware, configurations, scalability limits -- but about an external product.
First of all thank you very much for this very informative replies.
I would like to explain the entire situation in DETAIL so that you know EXACTLY what is going on and you might be able to give me some advice. That would be very great.
The company that I work for (I’m the only editor) is shooting a big project on several days. Something like 30 days!
30 one minute long films will be made out of this content. Now of course they don’t film chronologically so ie filming day 20 might contain footage for film A and for film X.
Now here is my task:
1) Copy all the footage onto a NAS. Our technician installed a 110 TB NAS for this job. I’ve got a SanLink 2 box hooked up to my fire wire which is connected to a glass fiber cable. I’m not a technician just an editor so I do not know all the technical details but it’s a gigabyte connection and should work really fast. About backups: Two hard disk drives of that NAS can „die“ before we run into serious troubles so yes I just park it all on the NAS because I do not have anywhere else to „double“ the files.
2) Import all the footage into FCPX. It will ALL BE 4K filmed with a RED EPIC, FFS5, FFS7.
Now my end 2012 iMac clearly has playback issues with 4K footage SO should I just transcode to proxies and forget about optimized media? Because I don’t know exactly what „optimized“ means. If I understand it correct it just converts the footage to Apple Pro Res 422. But 4K 422 is still too much for my machine too handle. So I think Proxies would be the best solution.
3) Worst part: Since it’s TOO MUCH to edit I will have to hand out the footage to 2 free lance editors who aren’t even located here.
The idea is: Editor 1 gets to edit film A,B,C… Editor 2 gets to edit film X,Y,Z and I also get to edit films.
My idea was: Create ONE master library that contains ALL the footage because I’ve got the storage for it.
Create Events for each day and import the footage by day. Once I’ve got events like DAY 1, DAY2 with all the footage inside FCPX I will look at the shoot list and look at what films have to be made (ie Film Horse Riding) and create events ALL NAMED like the films that need to be done. A good idea would be to give each film a NUMBER so that I’ve got event „01 Horse Riding“.
4) Get an external hard drive, create a library on that hard drive. Go into my master library and copy the event of film 1,2,3 onto the external hard drive.
After the freelance editor is done he gives me an XML and I relink my media.
OH one more thing: Normally my footage is always like this: Folder: Day 1 - Sub folders: Card 1, Card 2, Card 3.
I like the fact that keywords get their information from finder folders. FCPX doesn’t just import folders the way they are (or am I doing something wrong). Now of course- and I hadn’t thought about it yet - there will be a conflict because I will end up with MANY Card 1, Card 2, Card 3 folders from all the different days and FCPX (which makes sense) will merge all the footage from all those different days together into keyword collection Day 1, Day 2…using different events won’t save me from that problem. Of course I want to have event Day 1 with it’s 3 subfolders or keyword collections so I think the best solution would be just to rename he folders in the finder to: „Card 1 Day 1“ that way I can import a whole bunch of folders and files without having to worry that files won’t be where I expect them to be.
Any other idea for this? Thanks again.
Editing 4K over gigabit ethernet isn't going to work, no matter what machine you've got. The proxies might be okay, but transcoding will take ages, as will the final renders. I wouldn't bother generating optimized media, it won't help.
Options could include a direct-attached Thunderbolt RAID, with the NAS for nightly backups... or, if your NAS supports the option, maybe a 10GbE connection using something like this: http://www.sonnettech.com/product/twin10g.html
Your iMac should be okay with the proxy workflow, though I haven't worked with such a large amount of RED... someone with more experience in that realm would be more help than I can be.
As for the folders - I'd recommend day first, camera designation, then card number (Day01_CamA-01).
I replied to the last post (can't we edit our own replies anymore?)... I missed the part about the SanLink 2 further up - it supports the 10G connection I was talking about, so I suspect you're getting better that gigabit speeds.
[Doug Metz] "I replied to the last post (can't we edit our own replies anymore?)...
Had this happen and stumped me for ages. If the problem persists, try replying to your own post, then scroll up to the post you're replying to and you will see an edit option.
Jeff Kirkland | Video Producer & Cinematographer
Hobart, Tasmania | Twitter: @jeffkirkland