FORUMS: list search recent posts

Relinking

COW Forums : Apple Final Cut Pro X

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Oliver Peters
Relinking
on Dec 31, 2016 at 7:44:22 pm

Didn't this used to work better?

I'm in FCPX3 working on a project on my main machine. All media and library are in a single folder on my external drive at the root level. Media is external of the library, but all in this folder. I copy the entire folder to another drive in order to work away from home on my laptop. Still all at the root level of that new drive. So, the only thing different in any of the paths is the volume name.

I seem to recall that in the past, I would be able to launch the library from the new volume and FCPX was smart enough to relink everything. I may be wrong, but in any case, it no longer does that. You have to go through the relinking routine. BTW - that's take a lot longer than it should.

- Oliver

Oliver Peters Post Production Services, LLC
Orlando, FL
http://www.oliverpeters.com


Return to posts index

Noah Kadner
Re: Relinking
on Dec 31, 2016 at 9:02:08 pm

Yeah I would just take the actual drive- multiple copies of media is a headache. Perhaps one of the drives is formatted differently.

Noah

FCPWORKS - FCPX Workflow
FCP Exchange - FCPX Workshops
XinTwo - FCPX Training


Return to posts index

Oliver Peters
Re: Relinking
on Dec 31, 2016 at 9:17:53 pm

[Noah Kadner] "multiple copies of media is a headache"

That needs to be fixed. Taking the actual drive is not an option.

- Oliver

Oliver Peters Post Production Services, LLC
Orlando, FL
http://www.oliverpeters.com


Return to posts index


Noah Kadner
Re: Relinking
on Dec 31, 2016 at 9:32:47 pm
Last Edited By Noah Kadner on Dec 31, 2016 at 9:34:31 pm

Not sure a "fix" makes sense there. In most use cases you wouldn't want FCPX to automatically relink to a completely different set of media on a new drive just because it happens to match missing media in the Library. A "force" relink step is a more safe/explicit action.

Noah

FCPWORKS - FCPX Workflow
FCP Exchange - FCPX Workshops
XinTwo - FCPX Training


Return to posts index

Joe Marler
Re: Relinking
on Dec 31, 2016 at 10:04:59 pm

[Oliver Peters] "I seem to recall that in the past, I would be able to launch the library from the new volume and FCPX was smart enough to relink everything. I may be wrong, but in any case, it no longer does that. You have to go through the relinking routine. BTW - that's take a lot longer than it should."

I don't recollect it ever doing that. However if you rename the volume it should not require relinking at all. Maybe you are thinking about a case where the volume name was the same? However I did some tests on 10.3.1 (see below) and relink of a large library does seem slow, although I did timed testing on earlier versions.

Relink is supposedly faster if constrained to a folder tree vs searching a large drive containing lots of non-video assets. However my tests didn't indicate that much difference.

I just tested a relink on a large project we shot last week -- 2.3 terabytes totaling 199,000 total files and about 2100 4k video files, totaling about 50 hr. On a top-spec 2015 iMac 27 with media on a 16TB Thunderbolt 2 RAID-5 array, a full relink took 13 min to locate the files and about 60 sec to relink the 2100. However I had a lot of time lapse stills on that same volume (thus the 199k files) which weren't part of the video data and that probably slowed down the file location search.

I then repeated the relink test, first separating out the video only files to a separate folder tree. But this only improved the relink speed to about 10 min, which is still a long time for 2100 files in their own folder tree.

The FCPX code doing the file location does not manage the UI well, especially during the initial "Looking for matching filenames" phase. The UI is unresponsive for long periods of time and could give the impression the process is irrevocably hung. During this phase it doesn't do any I/O and each CPU core is at about 50%. The lack of I/O coupled with fairly low CPU could indicate an internal bottleneck, possibly serialized access to certain data structures -- IOW Amdahl's Law.

The UI then enters the stage of "verifying files for compatibility". This can take several minutes on a large library, even on a fast machine and disk subsystem. During this phase the UI updates some (although still poorly) and the progress bar can misleadingly stay on one file.

It then starts doing a lot of small I/Os -- the trace utility bitesize.d indicates an I/O histogram with distribution centered around 8k bytes. This might be FCPX using the Core Data SQLite database to update or build some table of the updated file paths, but I'm only guessing. But the initial lengthy CPU-oriented phase is not I/O limited.

I think for your case the easiest and fastest solution is just rename the volume using Finder>Get Info. If that is not possible then you'll have to relink it.

I don't have time to test it on my SSD RAID array but I don't think that would improve it much because the majority of the waiting time is not I/O constrained. It appears they need to improve the algorithm.


Return to posts index

Joe Marler
Re: Relinking
on Dec 31, 2016 at 10:13:31 pm

Sorry, meant I did NOT do timed relink testing on earlier FCPX versions, so I don't know if the current performance is worse or not. However it subjectively does seem kind of slow. 10 min to relink 2100 files in their own 1.5 TB folder on a Thunderbolt 2 RAID array seems a bit long. That said, relinking at least seems very reliable.

The ideal workflow would avoid relinking by renaming the volume and/or folder but that's not always possible.


Return to posts index


Oliver Peters
Re: Relinking
on Jan 1, 2017 at 2:10:01 am

[Joe Marler] "10 min to relink 2100 files in their own 1.5 TB folder on a Thunderbolt 2 RAID array seems a bit long."

Yep, that's slow. All I know is that it's considerably slower than Premiere Pro doing the same relinking task.

I presume, of course, that if the media were embedded into the library, then it wouldn't require any relinking. Unfortunately that just isn't practical.

- Oliver

Oliver Peters Post Production Services, LLC
Orlando, FL
http://www.oliverpeters.com


Return to posts index

Joe Marler
Re: Relinking
on Jan 1, 2017 at 3:22:47 pm

[Oliver Peters] "Yep, that's slow. All I know is that it's considerably slower than Premiere Pro doing the same relinking task.
"


I just did a smaller test of about 85 GB and 199 4k files, and Premiere CC 2017 is super fast at relinking -- however it is also somewhat "dumb". If you have lots of duplicate file names in different folders, as is common with Sony cameras (C00001.mp4, etc), it just grabs the first set of matching filenames -- whether they are correct or not. If you point it to the right subfolder, then it's both fast and correct since (a) you helped it, and (b) it was a simple case of a single folder name being different.

By contrast FCPX is doing some kind of more extensive match algorithm. I don't know if it is hash or CRC or what. This takes considerably longer, but it always finds the right files, even if there are many duplicate filenames scattered throughout the disk volume in a greatly differing folder structure than the original.

This enables you to point FCPX at the root and it will do whatever it takes to find the correct match. There are cases where the source/destination structure is greatly different and the recipient may not know exactly what all those changes were -- only that the media files are there, somewhere. I don't think Premiere can handle this situation, but I have not tested it extensively.

The problem with FCPX is you can't turn off that stringent file matching algorithm. Even in the above simple 85GB case, relink takes several minutes. There is also some kind of I/O constraint because the relink happens much faster on my Thunderbolt 2 SSD array than my spinning array. It also starts doing some kind of searching the moment you point it at the drive volume, without waiting until you navigate to the subfolder and press go.

In your case you have the exact same file structure and only the disk volume name is changed. It would be nice if you could somehow override FCPX and tell it "trust me, everything is the same so use a faster method".


Return to posts index

Oliver Peters
Re: Relinking
on Jan 1, 2017 at 5:10:28 pm

[Joe Marler] " If you have lots of duplicate file names in different folders, as is common with Sony cameras (C00001.mp4, etc), it just grabs the first set of matching filenames"

Correct. They also don't really have any sort of valid timecode - only a timestamp that get's interpreted as timecode. And not always in the same way by various apps. My regimen is to transcode any of these consumer formats to ProRes and then run Better Renamer to assign unique file names. Using these camera formats natively creates a lot of peril and should be avoided at all costs.

[Joe Marler] "The problem with FCPX is you can't turn off that stringent file matching algorithm."

Yes, it would be nice to have a "force relink" function like we used to have in FCP "legacy".

- Oliver

Oliver Peters Post Production Services, LLC
Orlando, FL
http://www.oliverpeters.com


Return to posts index


Joe Marler
Re: Relinking
on Jan 2, 2017 at 2:47:14 pm

[Oliver Peters] "My regimen is to transcode any of these consumer formats to ProRes.... "

Transcoding them to a non-consumer format which FCPX will not re-wrap is also a good idea for collaborative work when emailing FCPX "lean libraries" back and forth. That only works with "leave files in place".

FCPX will sometimes unpredictably re-wrap and copy certain consumer media to the library. The conditions for this are not documented. In the case of XAVC-S it may do this even though the import UI says "leave files in place". This in turn creates a situation where a huge library consisting mostly of "in place" files has been "poisoned" by inadvertent import of a few re-wrapped files. This grows the library too large for easy file transfer, yet there is no easy way to get those files out of the library. The consolidate function is all or nothing. With a 5TB "files in-place" library contaminated by a few of those re-wrapped files, you'd have to consolidate the entire 5TB to another location -- despite 98% of that already being externally managed. This is the only way to shrink the library to an emailable size. The solution is don't get in that situation in the first place, but this requires lots of testing since the situations causing it are undocumented.


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]