FORUMS: list search recent posts

FCPX horrendously slow to import clips or XML

COW Forums : Apple Final Cut Pro X

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Mathew Farrell
FCPX horrendously slow to import clips or XML
on Feb 17, 2017 at 9:53:06 pm

Hey folks,

I'm starting my first FCPX project. I'm putting together some long form docos and have some 60 hours of clips in at least three formats (including AVCHD files in MTS wrapper from a Sony).
The edit machine in question is a 2014 5k iMac 27" with 32GB RAM. Shouldn't struggle too much.

I started in Resolve, logging keywords for each clip before deciding this was a good project to throw at the fabled FCPX metadata Shangri-La. Installed the current version, 10.3.2

20 hours later I haven't gotten very far. FCPX hung several times trying to import all that media at once (leaving in location, not copying to the stupid library secret archive). I've broken it down into chunks to import, and some folders are fine, but the MTS clips don't look promising (loooong progress bars of Processing Files for Import that don't seem to progress).

Another problem I've got, assuming I can get my media in, is importing the keywording I've done in Resolve. As best I can tell, there's no facility to push the metadata to the clips in Resolve, but I can output a CSV file with it all in. I haven't tried it, but Shot Notes X sound like it can turn this CSV into an XML for me. The other solution I thought of was to drop all my clips into a timeline in resolve and kick out an FCPXML of them. that took 3 seconds, but FCPX hangs when trying to import that FCPXML.

It seems to me that FCP just can't handle any volume of MTS files. Can anyone think of any alternative workflows, or anything I'm missing before I toss it in trash and go back to Resolve?

Mathew Farrell
flowstate.com.au


Return to posts index

Noah Kadner
Re: FCPX horrendously slow to import clips or XML
on Feb 18, 2017 at 12:45:56 am

Couple things- 1- what is your disk drive type and speed? That would have a big impact on processing speed. 60 hours is a ton of footage
2 - MTS is by far the most processing intensive format to deal with. Also why do you need all 60 if it's coming from Resolve. If it was already rough cut there, why not first media manage in Resolve so that you're not dealing with such a massive project?

Noah

FCPWORKS - FCPX Workflow
FCP Exchange - FCPX Workshops
XinTwo - FCPX Training


Return to posts index

Joe Marler
Re: FCPX horrendously slow to import clips or XML
on Feb 18, 2017 at 2:26:41 pm

[Mathew Farrell] "...20 hours later I haven't gotten very far. FCPX hung several times trying to import all that media at once (leaving in location, not copying to the stupid library secret archive)....the MTS clips don't look promising (loooong progress bars of Processing Files for Import that don't seem to progress)....It seems to me that FCP just can't handle any volume of MTS files. Can anyone think of any alternative workflows, or anything I'm missing before I toss it in trash and go back to Resolve?"

These are bare .MTS files extracted from the AVCHD wrapper, else you wouldn't have the "leave files in place" option. The performance issues you describe are a known issue. MTS files should never be removed from the AVCHD wrapper, and if you do, FCPX does not handle it well.

The easiest and best solution is re-wrap the MTS files before import using EditReady: http://www.divergentmedia.com/editready

The rewrap is very fast and afterward enables lightning-quick import (ie a few seconds) with "leave files in place".

If you have any MTS files already imported, those should be deleted from the library and then re-import the rewrapped versions. If any bare MTS files are left in the library it can cause runtime performance problems, even if they are only a small % of the overall material.


Return to posts index


Mathew Farrell
Re: FCPX horrendously slow to import clips or XML
on Feb 19, 2017 at 9:56:28 am

These are bare .MTS files extracted from the AVCHD wrapper, else you wouldn't have the "leave files in place" option. The performance issues you describe are a known issue. MTS files should never be removed from the AVCHD wrapper, and if you do, FCPX does not handle it well.

HI Joe.
Removing from the wrapper is one thing, but I'm talking about leaving media on its source drive and not duplicating it all into the library file. Surely that's a global option, and nothing to do with the file format/codec. Are you talking about something different?

Cheers

Mathew Farrell
flowstate.com.au


Return to posts index

Joe Marler
Re: FCPX horrendously slow to import clips or XML
on Feb 19, 2017 at 1:04:57 pm

[Mathew Farrell] "Removing from the wrapper is one thing, but I'm talking about leaving media on its source drive and not duplicating it all into the library file. Surely that's a global option, and nothing to do with the file format/codec. Are you talking about something different?"

You need to :

(1) Remove any .MTS files from your library
(2) Rewrap the .MTS files using EditReady
(3) Import the re-wrapped files using "leave files in place". They will not be copied to the library.

This is super-fast and very reliable. Even though EditReady cannot remove the wrapper of MTS files "in place" and must make a new re-wrapped file, this is very fast. The re-wrapped files can then be imported "in place" -- typically within seconds. Your total processing time for all stages of import (inc'l manual re-wrap and file copy) will be vastly quicker.

Just make sure that you configure EditReady for re-wrap, not transcode. It can do either one.

There are two separate situations here, both involving AVCHD files:

(1) AVCHD media (even if within the AVCHD package) cannot be imported using "leave files in place" but FCPX itself will re-wrap those on import and copy them to the library. FCPX performance and stability should be OK. If you don't want them in the library, you'll need to re-wrap them yourself using EditReady prior to import.

(2) Bare .MTS files removed from the AVCHD package can be imported "in place" but this is very slow and can cause I/O performance problems for FCPX thereafter. It may be repetitively re-wrapping each file upon each reference. It does not handle it well, and that is the scenario you are facing. Unfortunately this is not documented in any KB article or Tech Note.

Normally FCPX handles various camera native files very well and you can get good editing performance using "leave files in place". However bare .MTS files removed from the AVCHD package are an exception. You have a very easy and straightforward solution for this, which will continue to be useful for any subsequent .MTS files you are given.


Return to posts index

Mathew Farrell
Re: FCPX horrendously slow to import clips or XML
on Feb 21, 2017 at 10:06:33 pm

Much obliged Joe. Thank you for taking the time to make the MTS issue clear.

Downloading Editready as we speak.

It's pretty amazing that the AVCHD issues aren't documents and made clear by Apple - the difference between In Place and In Library are huge, and can really mess up a workflow fast if it's not transparent.

Mathew Farrell
flowstate.com.au


Return to posts index


Bill Davis
Re: FCPX horrendously slow to import clips or XML
on Feb 18, 2017 at 7:52:24 pm
Last Edited By Bill Davis on Feb 18, 2017 at 10:11:00 pm

Also let me guess...

Are you one of those guys who decide to start "running" - so they go directly to sign up for a marathon?

I ask because even if you're EXTREMELY healthy and have lots and lots of other great strength and conditioning habits - it's just a REALLY bad idea to dive into a HUGE specialized undertaking - as your very first foray into a new form.

For most of us, it's what we learned in dozens or even hundreds of smaller jobs - that we bring to the table when we finally tackle something really big with hundreds of hours of source content.

Sure it can be done.

But starting with little to no basic knowledge of the landscape of a typical marathon - is NOT a good strategy to make it to the finish line unscathed your first time out.

Just my 2 cents.

Creator of XinTwo - http://www.xintwo.com
The shortest path to FCP X mastery.


Return to posts index

Mathew Farrell
Re: FCPX horrendously slow to import clips or XML
on Feb 19, 2017 at 9:50:58 am

Thank you for the thoughts and ideas guys. I like your Marathon analogy, Bill. Fair point. I'm disappointed that FCPX is struggling so badly with this, whereas Resolve hasn't batted an eye. Just because I'm disappointed doesn't mean I'm naive though - I'm well aware that different code has favoured and feared codecs, and it's also completely possible that some simple setup options would change matters.

The MTS files are unwrapped. It's how they arrived. I'll look into rewrapping. Good tip.
I'm not a fan of them, but they're alas a fact of life with this and many other projects these days. The land of a thousand codecs...

I'd have to check on the media drive's speed. It's connected via USB3.0

RE the Resolve to FCPX question - no cutting has been done. I spent a day reviewing footage and logging keywords in Resolve before going "Urgh, maybe this is the project to investigate FCPX on" because it needs a lot of media management. Resolve is way better than Premiere on this front, FCPX is supposed to be better still. I haven't got that far though...

Again, I appreciate your analogy Bill, but what part about the project or workflow is unfit or poorly done? I can't see too many errors one could make importing an XML into a blank project. Sure, there's plenty to cock up later, but I haven't even been granted that chance yet.

Can anyone tell me what FCPX is doing when it is Processing Files For Import? That sounds mighty like making proxies or renders to me.

I don't mean to sound adversarial at all guys, I really do appreciate the help. Just trying to wrap my head around the unworkable performance.

Mathew Farrell
flowstate.com.au


Return to posts index

Claude Lyneis
Re: FCPX horrendously slow to import clips or XML
on Feb 19, 2017 at 6:05:33 pm

This is a good reminder about trying to avoid shooting in AVCHD, if possible. It was useful when cameras had internal hard drives, but for the most part, camcorders have moved beyond that.


Return to posts index


Mathew Farrell
Re: FCPX horrendously slow to import clips or XML
on Feb 22, 2017 at 10:36:52 pm

Thanks for the thoughts guys.

Joe, I reckon you nailed it - Rewrapped all my MTS files and replaced them. Things are WAY smoother now. I think you're on the money when you say FCPX tries to rewrap them on the fly.
It seems odd to me that something like FCPX gets it knickers in a knot over the file wrapper, but can handle the codec passably. I would have though two or three lines of code would bypass whatever the wrapper is doing, but this is completely conjecture on my part - I don't know how that side of the formats work.

After rewrapping, the filenames remained the same, but the extension (.MTS to .MOV) was changed. I replaced the MTS files with the MOV files in the original media locations, hoping the break the links and reconnect to my rewrapped footage. I could select a single clip at a time and reconnect them no worries, but I couldn't find any way to batch the relinking, presumably because FCPX couldn't find and exact filename match.
I would have preferred to relink, as I'd already done some keywording and media management, but in the end it proved easier to trash those clips from the library and just import the rewrapped ones afresh. As they were in the same directory structure, they slotted into that part of my media management fine.

As a bonus question, does anyone know a batchable way to replace files with a different format or codec, like I was trying to do?

In researching this problem, I came across advice to keep libraries under ~1000-2000 clips. This makes sense. The workaround for a "project" like this (7700 clips) is to split the clips over several libraries (no need to move or dupe the source files), and build out a timeline from several libraries. I might investigate splitting this library up down the line, but so far things a slow, but not the point of hampering my keywording mission.

Mathew Farrell
flowstate.com.au


Return to posts index

Joe Marler
Re: FCPX horrendously slow to import clips or XML
on Feb 23, 2017 at 1:48:17 pm

[Mathew Farrell] "It seems odd to me that something like FCPX gets it knickers in a knot over the file wrapper, but can handle the codec passably. I would have though two or three lines of code would bypass whatever the wrapper is doing

It is obviously technically possible to handle bare .MTS files without major problems because Premiere does it. This is some kind of internal inefficiency with FCPX. Provided you understand it exists and use the solution you found, it's no problem.

[Mathew Farrell] "After rewrapping, the filenames remained the same, but the extension (.MTS to .MOV) was changed...does anyone know a batchable way to replace files with a different format or codec, like I was trying to do?..."

It wasn't just the filenames changed, but the internal file structure changed. Examination of a .MTS file using Invisor before and after rewrapping shows the byte count is different, duration slightly different and multiple header fields are changed: https://itunes.apple.com/us/app/invisor-media-file-inspector/id442947586?mt...

From a human standpoint you know it's the same content and codec, but by the stringent criteria FCPX uses, it looks like a different file. In theory these could still be relinked and used, and more flexible relinking has been requested many times.

[Mathew Farrell] "...In researching this problem, I came across advice to keep libraries under ~1000-2000 clips. This makes sense. The workaround for a "project" like this (7700 clips) is to split the clips over several libraries (no need to move or dupe the source files), and build out a timeline from several libraries. I might investigate splitting this library up down the line, but so far things a slow, but not the point of hampering my keywording mission."

I think that recommendation came from Larry Jordan. This has been discussed several times and we don't know the basis. I have a 3.5 terabyte library with 7,000 clips, and it mostly works OK. It had major performance problems before I cleansed it of all .MTS files, but it's mostly OK now, but the media is on an 8TB Thunderbolt 2 RAID-0 SSD array.

Extremely large libraries can still have I/O performance issues because FCPX does a lot of 8k random I/Os to build and maintain thumbnails in the Event Browser. Since even the fastest spinning RAID array is not good at small random I/Os, the only solution for this is an SSD array, and even those are not infinitely fast on that I/O pattern. Apple really needs to improve this. You won't see this on any "top 50" FCPX feature requests because most people either don't work at that level or are unaware of the cause. Hence if Apple goes by the number of requests, they may never improve this.

If you are at 7,700 clips it might be a good idea to consider splitting that, but if your current performance is usable you could wait and watch. If you can afford it an 8TB SSD array is only about $3k, provided you use the OWC Thunderbay 4 Mini and 4x Samsung EVO 850 2TB.


Return to posts index

Mathew Farrell
Re: FCPX horrendously slow to import clips or XML
on Feb 23, 2017 at 8:45:09 pm

Thanks again Joe.

Yup, that was Larry Jordan. We get similar advice for Lightroom libraries, although for obvious reasons the file count is a lot higher (quarter or half a million frames, if I remember correctly?) I suspect the database architecture works quite differently. In fact, if video playback wasn't horrendous, I'd use Lightroom for all my media management, then kick out selections to Premy, Resolve or FCPX if it grabs me.

And to be clear, I was 'easily' able to relink to my rewrappered files, but I could only link one at a time - FCPX was clearly content that it was "the same", but it seems to me it couldn't automatically line them up due to the file extension.

Mathew Farrell
flowstate.com.au


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]