APPLE FINAL CUT PRO: Apple Final Cut Pro X FCPX Debates FCP Legacy FCP Tutorials

Best syncing workflow between Laptop and Desktop

COW Forums : Apple Final Cut Pro X

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Declan Smith
Best syncing workflow between Laptop and Desktop
on Oct 9, 2017 at 11:19:15 am

I am relatively new to FCPX having converted back to FCP from Adobe.

I have a multi camera project in FCP X which I am currently working with mostly on my laptop. On the laptop "version" I am using proxy footage and on the desktop, I have the space for the original media. My question in a nutshell is what is the best way to sync the project between the laptop and desktop? I work on the laptop doing most of the edits on the train etc, then when back in the studio want to transfer to the desktop for further work, then back to laptop when I need to be out and about.

I have tried exporting project xml and importing, but have found that this doesn't appear to be ideal as the project browser doesn't keep up with changes. I have also had a few problems with audio edits reverting back, so right now, my laptop is the master but I need to use the desktop for colour grading / sound mastering.

I don't have the space for the original media on the laptop and although I could use an external drive, don't really want to have to plug one in when I'm on the move (happy to use it to backup / sync of course!)

Would really like to know of a workflow that allows simple handoff back and forth between computers taking into account different footage setup (i.e. one proxy / one original).

Declan Smith
http://www.madpanic.tv
FCPX / Adobe CS6/ FCS3 / Canon XLH1 / Canon 7D / Reason / Cubase

"it's either binary or it's not"


Return to posts index

Joe Marler
Re: Best syncing workflow between Laptop and Desktop
on Oct 9, 2017 at 3:10:24 pm

[Declan Smith] "On the laptop "version" I am using proxy footage and on the desktop, I have the space for the original media. My question in a nutshell is what is the best way to sync the project between the laptop and desktop? I work on the laptop doing most of the edits on the train etc, then when back in the studio want to transfer to the desktop for further work, then back to laptop when I need to be out and about."

There are several ways to do this, depending on the specific requirements and your configuration.

One option is to export an event XML from the laptop machine and import that on the desktop. That will include not only the project but any metadata changes you made such as keywords, favorites, rejects, etc. Note you will get a pop up warning upon import saying there are already duplicate clips in the library. You answer "replace" to this. Before loading any XML it's a good idea to make up a manual backup copy of the library by shutting down FCPX, clicking on the library and selecting "duplicate". If something goes wrong you can just roll back to that.

Another option is keep the library on a portable drive. By default the proxies will be in the library. The procedure is easy: in the field edit on laptop and keep FCPX in proxy mode (In the viewer drop-down menu, View>Proxy). Then when you get home you connect the portable drive to your desktop and it will automatically connect to the original media.

However if your library is on the laptop itself then you'd have to copy the entire library from the laptop to the desktop to resume work. If your work is mostly sequential, say you defer color and audio work until the end then do that on the desktop, this might not be inconvenient to copy it once, which includes all the contained proxy files inside.

However if you are frequently changing between desktop and laptop and if you aren't using a portable drive, another option is use external proxies which keeps the library size small (typically < 100MB, often less than 50MB). Then when you get home you copy the library to the desktop, start FCPX and it should theoretically connect to the original media. However if the drive name is different this may require a relink. In my experience relinking when using external proxies is unreliable so this should be examined carefully before using it. A workaround is have the desktop and laptop both use the same volume name for the internal drive, if that's where your media is.

Also, the procedure for creating external proxies can be a bit complex. The entire procedure including various issues is described in Ripple Training's FCPX 10.3 Media Management tutorial:

https://www.rippletraining.com/products/final-cut-pro/media-management-in-f...


Return to posts index

Joe Marler
Re: Best syncing workflow between Laptop and Desktop
on Oct 9, 2017 at 3:32:10 pm

I forgot to add there is a bug in FCPX event XML processing which mandates that all media filenames be globally unique across the entire library. Many cameras (esp. some Sony NEX series) do not allow in-camera filename customization so will use repeating filenames each time an SD card is changed. Upon import FCPX appends a "uniquifier" but something still malfunctions during the import phase of XML processing on the destination machine.

Even if the media files are kept in separate disk folders (say by shooting day and by camera), after import to FCPX this creates a situation where loading an event XML will create spurious duplicate clips.

The only solution is maintain absolute global filename uniqueness before importing media. That might be a good practice anyway but it should not be required. If the media is in folders by shooting day and by camera, FCPX can automatically keyword ingested files by those folder names, so it's easy to find using the FCPX query tools -- regardless of filename.

Since you must assume that your material might later be combined with content from other projects, the global uniqueness requirement becomes more difficult. They must not only be unique within the event but unique across all libraries for all time, at least until Apple fixes the bug. I use the tool "A Better Finder Rename" before import, append a sequence number to each file, keep track of the highest number so each ingest batch uses sequential sequence numbers:

http://www.publicspace.net/ABetterFinderRename/

If your media also includes metadata files with matching filenames, those also must be renamed. However I think some cameras embed the original media filename inside the metadata files, so this might require a special utility to rename the files properly. To reiterate, this is only required if you are loading event or library XML files and the on-disk media filenames are not totally unique.


Return to posts index


Declan Smith
Re: Best syncing workflow between Laptop and Desktop
on Oct 10, 2017 at 8:14:18 pm

Thanks for the comprehensive response to my query. I have tried with exporting the event to an XML from the laptop and importing it to the desktop and it seems to be working fine.

I think previously I was exporting on a project by project basis, but now I see the keywords and changes as expected. I have a little work to organise the external assets so they are consistent between laptop / desktop, but that for now is relatively easy.

I have also now tried removing the library and creating a new empty Library and imported my event XML from the laptop. A simple relink was all that was needed and this too appears to work. There are 4.5 hours to double check (conference), but the quick checks I've done seem to be ok.

I haven't gone back the other way just yet, but I will do so once the carbon copy clones on my laptop have finished, just in case!

Your other suggestion about naming is a good one. I have seen similar issues in other packages where determining which "MVI_1234.MOV" you want to relink can be problematic. It's good practice to have unique names.

I have looked at other training materials, but the ripple training does seem to be more comprehensive so I will have a look at theirs also.

I will update this thread if I come across any issues, but thank you once again.

Declan Smith
http://www.madpanic.tv
FCPX / Adobe CS6/ FCS3 / Canon XLH1 / Canon 7D / Reason / Cubase

"it's either binary or it's not"


Return to posts index

Joe Marler
Re: Best syncing workflow between Laptop and Desktop
on Oct 10, 2017 at 4:18:25 pm

[Declan Smith] "tried exporting project xml and importing, but...the project browser doesn't keep up with changes. I have also had a few problems with audio edits reverting back... "

I used event XMLs a lot and haven't seen cases where the project isn't updated when loading an event XML. However XML export/import of a multicam clip does not preserve the audio or video monitoring angle as selected in the angle editor. The selected audio in the angle *viewer* is preserved. For me it's a minor issue. It might be by design -- maybe they consider the monitoring angle to be application state data not video metadata.


Return to posts index

Declan Smith
Re: Best syncing workflow between Laptop and Desktop
on Oct 10, 2017 at 8:16:37 pm

Hi Joe,

I think I was exporting at the wrong level before, i.e. at the project by project level. Exporting the event seems to be working nicely.

Many Thanks

Declan Smith
http://www.madpanic.tv
FCPX / Adobe CS6/ FCS3 / Canon XLH1 / Canon 7D / Reason / Cubase

"it's either binary or it's not"


Return to posts index


Fabrizio D'Agnano
Re: Best syncing workflow between Laptop and Desktop
on Oct 11, 2017 at 9:07:33 am

Hello.
You sure have your reasons not to use an external drive, but it seems a great option to me. I use an external USB3 box containing an ssd drive (actually, a few of them, each dedicated to one series of projects). I keep my projects self contained inside a Library, with media also managed inside the library and not external. The drive is large and fast enough for my needs, and is bus powered, so I can connect it when working on a train or wherever. I always keep working on the same drive, connecting it to any machine running the same FCPX version, and for safety reasons only I sync the library via Sync Folders Pro onto a safer raid 5 array. I find this solution very practical and easy, all the media stays together in one container I can work on without really moving or touching anything, so the risk of damaging something is limited and I don't end up with a missing something that was not copied from one machine to another. If the RAID 5 was actually faster than the external SSD (and it seems it's not), I would work on that on after synch'ing with the ssd, and sync back (bidirectional) after the session (it is really fast if you're not adding a lot of new media).
Just my 2%

Fabrizio D'Agnano
Rome, Italy


Return to posts index

Declan Smith
Re: Best syncing workflow between Laptop and Desktop
on Oct 13, 2017 at 10:36:45 am

Hi Fabrizio,

I have a number of workflows depending on the project and how I will be working with it. The reason I am not using an external drive as the primary drive in this case is because of the fluidity of movement when I'm on the go and the time I can get to do editing. i.e. I am hopping on and off trains, into cafes, sat waiting for trains at the station etc. In these situations it is not wise to have an external drive plugged in for two reasons,

1. it could easily be pulled out or knocked
2. I need to move quickly.

My workflow for this is therefore "I have 5 minutes, open laptop, edit, shut lid, move on". A regular mechanical hard disk has physical risks in these situations, and SSD's have the risk of electrically being erased if unplugged accidentally.

Also, there is no need for me to have the full suite of large original footage on my laptop. It's a much faster workflow to edit with low res proxies, I never have to wait for rendering and there is no stuttering etc. I can still do primary grading on low res proxies using scopes, leaving the main grade to the desktop suite with calibrated monitors etc. So the only thing I need to move around between the computers is metadata, i.e. what's changed in the edit, keywords, etc. Yes, if I add additional assets like stills, I need to manage them myself outside the project.

The xml export/import was engineered with this very portability in mind and now I've been directed to export at the event level (rather than project) the world has become even more beautiful!.

For example you could email an xml to another editor if you needed to (especially if you compress it - the project I am working on is 10.3Mb xml -> 293K when compressed). So from the train, I could email all my changes to a colleague back in the office at a snap. All I have done to the project is a few cuts here and there, and a few metadata changes. No need to send a whole library around everytime, as long as external media is organised, which in my case is similar to the FCPX library concept in that it all lives under one root folder, making relinking very trivial.

There are merits in having everything rolled up in one library, but this in itself is just a folder structure, treated differently by MAC because of folder attributes, but a folder all the same. FCPX creates all sorts of files for internal use, rendering etc, which are all very useful, but superfluous and throwaway. For me, I manage my media externally from the editing package and manage my backup strategy accordingly. This has the added advantage that the media can be used in more than one package directly without having to worry about the internals of any of them.


Slightly off topic but worthy of note to discuss disks.

With regards to RAID5, I take it you have never had a failure? I never use RAID5 for anything anymore. A single disk failure means that the array still works yes, but at reduced performance, and the time it takes to rebuild after a failure is astronomical and degrades performance during the rebuild (and carries it's own risks). Once an array is in a degraded state you should immediately take steps to repair the array!

Then when you have more than one disk fail, which is more common than you think because most RAID5 arrays are built from disks that were made at the same time, then a lot of people get a shock because they use it mistakingly as a backup solution then realise they have lost everything! This may not happen very often, RAID5 has mostly good press, but when it does go wrong, it can be potentially catastrophic.

SSD's are fantastic for their speed, I use them in RAID0 configuration in my desktop and my laptop disk is also an SSD. I wouldn't be without them, BUT, they should be treated as unreliable data stores. Why? because if something goes wrong with an SSD you will lose everything, no exception, everything, no recovery. A simple power glitch can do this. This is due to the technology used.

Always separate operational use / responsiveness from backup / data security. I've never had an SSD failure in my computer (yet), but I have had it happen to SSD's used to capture footage (on more than one occasion). Think of them as volatile, non-volatile RAM devices!!

My strategy is a 3 backup system with one copy off site, and that is simple mirroring with contingency (I use carbon copy cloner which has a safety_net feature, so even though each time I clone, if I did accidentally delete or change files between clones, they are preserved , a bit like time machine but a 1000 times better!).

I use RAID0 on most things for performance reasons. This brings the added danger of losing everything in the event of a failure, just like SSD's, but the 3 backup system works around this also.

Simple mirroring requires no rebuilds, no performance hits, and total reliable data security. The pay off is that I have three times as much storage than I actually use, i.e. if I have a 6TB disk, then I have two other 6TB disks for backup which gives me a total of 6TB usable storage from 18TB, but it's secure and reliable and a life saver!

With the Carbon Copy Clone approach, if my main computer disks were ever to fail, and I need my computer in it's entirety to work right now because I have an urgent project, then all I need to do is boot off the backup and carry on working, leaving the replacement and re-cloning to when I have time. Things never break conveniently.

This 3 backup system is tried and tested. I also used it for DIT workflows when offloading footage as well as general backup etc. It has saved me many a time!

Declan Smith
http://www.madpanic.tv
FCPX / Adobe CS6/ FCS3 / Canon XLH1 / Canon 7D / Reason / Cubase

"it's either binary or it's not"


Return to posts index

Joe Marler
Re: Best syncing workflow between Laptop and Desktop
on Oct 13, 2017 at 11:43:17 am

[Declan Smith] "I never use RAID5 for anything anymore. A single disk failure means that the array still works yes, but at reduced performance, and the time it takes to rebuild after a failure is astronomical and degrades performance during the rebuild (and carries it's own risks)....Then when you have more than one disk fail, which is more common than you think because most RAID5 arrays are built from disks that were made at the same time.....if something goes wrong with an SSD you will lose everything, no exception, everything, no recovery....I use RAID0 on most things for performance reasons."

You have a lot of good advice here. I also don't use RAID5 anymore for the same reasons. I formerly said I'd never use four-drive RAID0, but for 4k or uncompressed media this makes sense, and I use it all the time now. Of course the FCPX libraries are backed up elsewhere.

The media storage is very large and we always have redundant copies anyway, either on the original portable offload drives or another RAID array. It doesn't make sense to take the performance penalty and size penalty of RAID5 when the media is already backed up and it isn't changing much (if any). The RAID5 array (despite the redundancy) must itself be backed up because it can have a chassis failure, application database failure, filesystem failure or human failure. If it's backed up anyway and the content isn't changing, why use RAID5? The only reason is ability to continue work if a single drive fails. If a 2nd drive fails before the rebuild finishes, the entire array is lost.

However, in that case the RAID5 array becomes very slow during the rebuild phase, so the ability to continue work can be marginal. Also other drives in the array are then at statistical risk due to manufacturing batches and shared environmental issues. IOW if one drive fails, then others are more likely to fail. Higher RAID forms mitigate this but at even greater expense, storage or performance cost, and they do nothing to help filesystem, chassis, database or human failures.

If a four-drive RAID0 array fails, I simply switch to the backup array which is loaded via Carbon Copy every night. The media itself does not change during the day. Manual switchover to the backup array takes two minutes. This does require a backup array but the main array (whether RAID0, RAID5 or anything else) must be backed up anyway. If it's backed up to an array with roughly comparable performance, that an be switched to the main array.

If it is backed up to slower media like LTO tape or bus-powered portable drives, the backup is cheaper per MB but recovery time becomes a significant issue. A good backup that nonetheless takes many hours to restore entails significant downtime.

Your comments about SSD are also correct. SSD can fail like any other drive and must be backed up. While SSD reliability in general is better than spinning drives, they still have a significant failure probability. In some past studies, SSDs failed at a higher rate than some spinning drives:

http://media.bestofmicro.com/4/T/302141/original/ssdfailurerates_1024.png


Return to posts index


Fabrizio D'Agnano
Re: Best syncing workflow between Laptop and Desktop
on Oct 13, 2017 at 6:18:17 pm

Hi Declan.
You raise a lot of very interesting points. It's very true that different workflows can fit different needs.
I used to work with external media managed in folders until a couple of years ago, then switched to self contained libraries. I don't need to exchange projects with other people, so I find this procedure easier and more error proof for me.
About the RAID5, I agree it's not a safe solution as it is. I had to rebuild a system once during a major reorganisation of years of footage, and it was painfully slow and scary. Once I had a problem with the hardware, so that the only solution seemed to find and buy an identical unit and put the drives inside. The unit eventually got fixed, but I learnt the lesson. I like to use one RAID5 ARRAY as the hearth of my editing suite because it's large enough so that it can contain the last two or three years of projects and media, and allows me to search indexed clips throughout years of footage. An example could be finding clips of a particular jelly fish I may have shot in several different occasions in the past three years. Since I don't use it for editing, I trade the extra speed and space I'd get if it were a RAID0 for that bit of extra safety. I copy the needed footage on the external drive, and keep editing on that. The finished and delivered projects go from the external SSD to two more separate drives, so I keep three copies anyway, while the current projects that are on the external SSD's have a back up on a RAID1 system, so it's actually four copies running, one of them on RAID5. So if a SSD fails I can just copy back the libraries from the RAID5. Should the devil be on my shoulders and burn one drive in the RAID5 array while I'm rebuilding the SSD, I still have the RAID1 to rebuild copies on the ssd from and keep on working, before restoring the RAID5, or trying to. I totally agree that at least three copies, one in a different location, are a bare minimum for critical data.

Thank you for your insight!
Fabrizio

Fabrizio D'Agnano
Rome, Italy


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]