Apple Final Cut Pro X Forum
Keep the camera originals or not - what say you?
Keep the camera originals or not - what say you?
by Tangier Clarke on Mar 8, 2019 at 6:43:22 pm

Folks, this may be a non-starter and seemingly ridiculous question to even ask, yet I am curious what your thoughts are and why.

As with many productions I sometimes we run up against budgets and the cost of storage for backing up and actually working (post-production).

I am working with content from a Sony FS7 shot with a Sony log profile which is outputting mxf files in a card structure I don't fully understand - being that FCP X (and likely other NLEs) just parse through to get to the content.

My question is - after rewrapping content to Quicktime .mov files, copying to a drive, importing to your NLE or what have you, do you keep or feel the need to keep the original card-structured folders. Essentially I have duplicates of now of the same content. The original structure on a hard drive and the rewrapped.

Admittedly not having spent the time to take an analysis approach to the folder structure and files for each clip (full of XMLs, UserData, sub folders, etc.) is there an incentive to keep the originals or is it safe to use the rewrapped files as the new masters and back up those?


Re: Keep the camera originals or not - what say you?
by Craig Seeman on Mar 8, 2019 at 9:15:37 pm

I always keep the camera originals. I use KeyFlow Pro to keep track of my archives. One concern is one never knows what permutations NLE development will go through (EOL for some and new ones arising from the ashes). I just feel safer in that I might be able to pull something up 10 years from now with the camera originals.

For example, I think it's more likely Sony will have some form of Catalyst permutation that will read their old formats than NLE developers handling legacy formats (especially as they leave or enter the market).




Re: Keep the camera originals or not - what say you?
by Tangier Clarke on Mar 8, 2019 at 9:25:22 pm

Thanks Craig. That is something I was thinking about - the notion of .mov files being supported down the road and/or this FS7 mxf format and folder structure being supported. In general I would agree that one should always keep the camera originals. By the time I start editing, I've rewrapped, custom named, and sometimes restripesd the timecode of my video clips based on the LTC audio from lockit boxes. Our editorial and archiving will reference these files. The process to get the clips into this position would never be retained with the original clips and one would have to redo this process to get there. Essentially it boils down to just sucking it up and having the originals backed up, the originals rewrapped and processed for editing backed up. It'd basically be having to accommodate double the storage needs for the same content.


Re: Keep the camera originals or not - what say you?
by Tangier Clarke on Mar 8, 2019 at 9:31:15 pm

I guess this begs another question: Is the camera original any more future proof than the rewrapped version of the same file? Granted we're not talking transcoded files either.


Re: Keep the camera originals or not - what say you?
by Craig Seeman on Mar 8, 2019 at 10:38:12 pm

[Tangier Clarke] "Is the camera original any more future proof than the rewrapped version of the same file?"

I'd say yes. Keep in mind that it's not only codec or container compatibility but how well supported is the specific codec container combination. While the codec and container may be widely supported the combination might not be supported if a specific NLE developer was using the combination for their NLE. If other NLEs aren't supporting the combination you'd really want the ability to rewrap the camera original for the new NLE you're using at the unknown future date.




Re: Keep the camera originals or not - what say you?
by Joe Marler on Mar 9, 2019 at 2:30:41 am

[Tangier Clarke] "Essentially I have duplicates of now of the same content. The original structure on a hard drive and the rewrapped....is there an incentive to keep the originals or is it safe to use the rewrapped files as the new masters and back up those?"

Tangier, as always a good question. The problem is the original camera files cannot be relinked to your library because you imported from rewrapped files. The filenames differ. In some cases you can change the suffix and it will relink but that is tricky. If any other part of the filename is changed it won't relink.

Obviously you can frame match each clip in the project by eye or maybe timecode offset into the clip but that is very tedious.

If the files were wrapped before import with something like EditReady, the mov files are probably OK long term. There is a small possibility some future change in how metadata is handled might require re-import from the camera files.

I think there's probably a greater chance of a future request for some slight change in the project, then you need to reload it (plus media) and make that change. In that case having the camera files wouldn't help since you can't relink to those.

If the material isn't that big you could keep both. However for archival storage you need at least two separate copies. Any single disk or tape can fail at any time. So the feasibility/risk calculation should include this 2x penalty, which would be 4x for both camera and re-wrapped files.

I would personally rewrap those using EditReady, rename the files according to our system, import then discard the camera files. If the material was of extreme value I might also keep the camera files.


Re: Keep the camera originals or not - what say you?
by Craig Seeman on Mar 9, 2019 at 2:39:52 am

[Joe Marler] "I think there's probably a greater chance of a future request for some slight change in the project,"

I've had a few clients that were creating entirely new projects and wanted shots that may not have been used in the original. Sometimes they're looking for things left "on the cutting room floor" as it were. It only happens occasionally but it was far easier for me to import from the camera masters into FCPX than to try to resurrect and FCP legacy project. The new project wasn't dependent on the legacy one.




Re: Keep the camera originals or not - what say you?
by Bret Williams on Mar 9, 2019 at 3:11:49 am

I Rarely rewrap. Just drag the fs7 card structure directly to an event. Edit directly with the mxf originals.

_______________________________________________________________________
http://BretFX.com FCPX Plugins & Templates for Editors & Motion Graphics Artists
Hang Tag http://bretfx.com/product/hang-tag
Overshoot Text http://bretfx.com/product/overshoot-text/
Outliner http://bretfx.com/product/outliner/
Clock Maker http://bretfx.com/product/bretfx-clock-maker/


Re: Keep the camera originals or not - what say you?
by Tangier Clarke on Mar 9, 2019 at 3:31:43 am

Bret how are you doing this? From the import window when I have an FS7 card ready to bring in, the option to leave in place is not an option; even with the cards are effectively folders on a hard drive where one would think FCP X would allow it. I am forced to copy to library. I'll be honest that I found this a little strange. If I was importing from the cards then I could understand FCP X behaving this way, but from folders on a drive, FCP X still treated the media like removable cards for me.

FCP X 10.4.5


Re: Keep the camera originals or not - what say you?
by Bret Williams on Mar 9, 2019 at 3:34:53 am

Just drag the folder directly to the event. I open the import window like once a year. If you have spanned clips they will come in separately, but that’s not such a big deal.

_______________________________________________________________________
http://BretFX.com FCPX Plugins & Templates for Editors & Motion Graphics Artists
Hang Tag http://bretfx.com/product/hang-tag
Overshoot Text http://bretfx.com/product/overshoot-text/
Outliner http://bretfx.com/product/outliner/
Clock Maker http://bretfx.com/product/bretfx-clock-maker/


Re: Keep the camera originals or not - what say you?
by Tangier Clarke on Mar 9, 2019 at 4:09:12 am

Will give it a shot. Though doing some reading and depending on usage of effects, color, just simple edits etc. my understanding is that although FCP X plays the MXF files just fine, that there are circumstances when not to use the raw mxf files. I seldom used the import window either. I always just drag to the event. I need to double check that FCP X behaves differently when using either method in terms of copying or leaving in place. I think it does though.


Re: Keep the camera originals or not - what say you?
by Bret Williams on Mar 9, 2019 at 4:14:27 am

It completely does. It’s even in the manual. Some formats really suck when not rewrapped. some Sony AVCHD I think. But if you ever have performance issues you just hit the transcode button and it’s then working off ProRes.

As long as you’re not working off Xeon, all these h264 formats are handled great by a MacBook or iMac natively. I do lots of layers and effects, plugins, compounds inside compounds inside compounds. All good. when you drag in, your import settings are what’s in the import settings in FCPX prefs.

_______________________________________________________________________
http://BretFX.com FCPX Plugins & Templates for Editors & Motion Graphics Artists
Hang Tag http://bretfx.com/product/hang-tag
Overshoot Text http://bretfx.com/product/overshoot-text/
Outliner http://bretfx.com/product/outliner/
Clock Maker http://bretfx.com/product/bretfx-clock-maker/


Re: Keep the camera originals or not - what say you?
by Tangier Clarke on Mar 9, 2019 at 5:31:40 am

I am on a Mac Pro (2013-Black trashcan with Xeon processors) and a 2016 MacBook Pro. The MP handles all my footage just fine. It's just horrible when it comes to compressing or transcoding H.264. That's where my MBP shines and even my old 2012 MBP would run circles around my MP due to Intel QuickSync. The FS7 MXF files at 4K on a 4TB LaCie mobile rugged RAID play incredibly fluidly over 1080p HD from a C100. I can scrub though these MXF files (rewrapped or not) with no lag. Multicam is pretty decent too with two FS7 cams and no proxies yet. I count this to the XAVC-I intraframe codec rather than interframe.


Re: Keep the camera originals or not - what say you?
by Tangier Clarke on Mar 9, 2019 at 3:28:38 am

Craig I would only see this as a real issue if when importing or pre-processing clips one picks and chooses the clips to import. I never do that. I always pre-process and or import all of the clips of every shoot. I don't trust myself nor my associates to keep track of what we did and didn't import. This way, when we rewrap or transcode clips depending on your process, we know we have it all. If a client want's something that was left, we'd still have it. It'd simply be in mov form and not camera card original form.


Re: Keep the camera originals or not - what say you?
by Craig Seeman on Mar 9, 2019 at 3:34:09 am

[Tangier Clarke] "It'd simply be in mov form and not camera card original form."

But another NLE may not like the camera codec in a mov container. It's too hard to predict where the industry will years forward from now.




Re: Keep the camera originals or not - what say you?
by Joe Marler on Mar 9, 2019 at 1:21:19 pm

[Craig Seeman] "...wanted shots that may not have been used in the original. Sometimes they're looking for things left "on the cutting room floor" as it were. It only happens occasionally but it was far easier for me to import from the camera masters into FCPX than to try to resurrect and FCP legacy project. "

In my case it's not the legacy project but the legacy library that's important. The last doc I worked on had 125 multi-cam interviews, 151 camera hours of material, 62 hrs external multi-channel audio, 3,133 clips totaling about 10 terabytes. Weeks of effort by multiple assistants went into rating and keywording the material. This enables later rapid retrieval during post -- and afterward.

After delivery, if a subsequent request comes in to find a certain camera angle from a certain interview or to find additional b-roll to cover dialog, for me it's usually easier to leverage the library which is already highly organized. This is especially for cases where sync'd multicams or sync'd external foreign language audio is required -- that's already done inside the library. If I kept only the camera files I couldn't relink to those so the massive investment in the library would be lost.

In Tangier's case, another complication is he probably cannot import the MXF with "leave files in place". That can be a major issue if dealing with lots of material. It makes the library huge, which makes file-level library backups difficult. You can alter library properties to store that externally but that media will be in the reorganized FCPX folder structure and bear little resemblance to the original.

I like doing most of the data organizing inside FCPX, but I do a small amount outside before import -- rename files, use a basic folder structure based on shooting day, operator+camera type, etc. Tree-oriented camera media is a dilemma. If you try to rename those in Finder, it may run afoul of embedded metadata storing the original filename. Some manufacturers have proprietary ingest tools to handle this. Or if you copy the bare video files out of the tree to obtain "leave files in place", it might cause other problems.

Whether the files are re-wrapped by FCPX during import or by EditReady before import, if those are not saved you cannot later relink, thus you can never reload the project OR library. This creates the "double save" dilemma Tangier is trying to avoid. You either trust the rewrapped files and write off the camera files or buy more disk drives or LTO tape and save two copies of camera media *and* two copies of the rewrapped media.


Re: Keep the camera originals or not - what say you?
by Bret Williams on Mar 13, 2019 at 5:12:30 pm

Ah but you can leave MXF in place. Just drag them directly from the finder to the event window. MXF is even supported by quick look these days.

_______________________________________________________________________
http://BretFX.com FCPX Plugins & Templates for Editors & Motion Graphics Artists
Hang Tag http://bretfx.com/product/hang-tag
Overshoot Text http://bretfx.com/product/overshoot-text/
Outliner http://bretfx.com/product/outliner/
Clock Maker http://bretfx.com/product/bretfx-clock-maker/


Re: Keep the camera originals or not - what say you?
by Tangier Clarke on Mar 9, 2019 at 3:25:49 am

Joe, I almost exclusively use EditReady to pre-process all of my media to rewrap it to .mov files and also to create unique nomenclature for the files; at least for AVCHD content coming from a C100. Also, it's been my routine for many years to archive to to cloned SATA drives and then use NeoFInder to catalogue them. I agree, one will fail at some point.

In the case of the FS7 clips, FCP X actually retains more of the correct metadata importing it directly, rather than through EditReady. They're close, but FCP X is still a little better. Essentially FCP X is rewrapping it too. I am not having it transcode on import being that FCP X supports MXF anyway. I just pull the clips out of the library package, and can use the Finder to rename them and bring them back in. I really wish FCP X would bring back the FCP 7 feature to rename finder clips based on the names in the NLE since FCP X can do custom naming like EditReady; albeit it's bound exclusively to the FCP X database.

If there ever was a need to resurrect a project it for sure would be using the rewrapped media, not with the camera original. I am trying to avoid the 2x or 4x penalty too.

I hear what you're saying though. There really is no perfect answer. It depends on the importance of the content among other things.


Re: Keep the camera originals or not - what say you?
by Tangier Clarke on Mar 13, 2019 at 6:02:17 pm

Ok, folks I've been doing some testing, examining, playing around with these MXF files from the Sony FS7 and looking into the XML files in the card structure. Though DaVinci Resolve is not pertinent for what I am trying to find out, it helped to have another NLE to see how two different systems handled the same content, which in turn helped me verify (or not) certain things about the MXF and MOV files. Here's what I discovered:

Tools: FCP X 10.4.5 and DaVinci Resolve 15, MediaInfo app
Hardware: 2013 Mac Pro 3.5 GHz 6-Core Intel Xeon E5/ 64 GB RAM /Two AMD FirePro D500/ Thunderbolt 2 connected drives.
OS: mac OS Mojave: 10.14.3

Using FCP X import window rewraps MXF files to .mov with no option (even from hard) drive to "leave in place". Dragging in MXF files to FCP X event keeps MXF files as MXF and MXF files are allowed to stay in place inside their card structure on the hard drive.

Playback of MXF files in DaVinci Resolve 15 is much choppier than FCP X. Lots of green scrubbing frames as well in DaVinci Resolve. The rewrapped version of these files played back much smoother than their originals.

Using MediaInfo I noticed that the .mov FCP X created (due to using the import window) which was a rewrap it seems and not a transcode of the MXF retained all of the pertinent information and metadata that the MXF had and needed for playback and identifying the camera LUT used; in this case - Sony S-Log3/S-Gamut3.Cine.

I created a new library in FCP X and brought the .mov file in to see if FCP X had and/or would identify the LUT (meaning that it's truly with the file) and it applied the LUT right away. The LUT info was presented in the inspector in the Camera LUT fields and Color Profile. I didn't want to give FCP X a chance to use any potential previous database info on how to handle the file. To further confirm the LUT info was there I used DaVinci Resolve 15 and brought the same clip in. It showed flat. I had to apply the LUT manually. Behaviors between the two apps are different, but it seems nothing so far is lost.

The rewrapped MXF to .mov is an mp4 Quicktime inside a .mov. I am thinking that it has to be this way because because perhaps mp4 don't support PCM audio, but .mov files do. (I could be wrong on this).

The rewrapped .mov from the MXF seems to have all of the pertinent information needed to keep these files instead of the MXF files. I compared information using MediaInfo going line by line to review all of the file structure, metadata, and header information. I recognize this is not scientific.

If money were no object it wouldn't matter. We'd keep everything. As it stands that's likely what we'll do anyway for now, but in the end everything will be edited, archived, referenced, searched, and potentially one day unarchived using the .mov files. So I am keeping these findings in consideration in the case I do have to (dare I say it) ditch the Sony card structure originals.

I know there is no one size fits all solution.


Re: Keep the camera originals or not - what say you?
by Joby Anthony Jr on Apr 8, 2019 at 5:41:06 pm

Tangier, I know this is a few weeks down the road from your initial question, but you described the same dilemma we've wrestled with. It resonated.

For what it's worth, what I've ended up doing is saving the card originals on a couple of harddrives for a period of time. As the drives fill up, basically rolling off old cards as newer ones come in. That way I can always go back to a card original if I need to for any reason (e.g., are we missing a clip from the card, corruption somewhere in the handoff, rewrap, rename, moving, etc. process, and so on). I've probably got a year and a half worth of cards I can go back to. Comes in handy from time-to-time, especially as a double-check. But ultimately relying on the re-wrap re-names as the master files down-the-line, for many of the reasons mentioned by others.

I also appreciated your testing and reporting on re-wrap/don't re-wrap. Bret's info about treating mxf files was news to me. But glad to know that at least your experience seemed to reveal no harm either way. Might help me re-think some of our workflow.

And to Joe's point about future-proofing the library as maybe just as or more important. Yes!! I discovered much later than I should have that little tidbit coming off 10.0 several years ago. Once discovered, did a 30-day panic to convert everything. I pay closer to attention to these things now!

Joby.


Re: Keep the camera originals or not - what say you?
by Tangier Clarke on Apr 8, 2019 at 10:26:14 pm

Joby, I am glad to hear it had some value. I am backing up all the cards to a RAID we have at the office, but I've rewrapped and restriped the Tentacle and/or Ambient box timecode (using LTC convert) of those clips, organized them and synced them (using Sync N Link X and FCP X audio syncing) onto another drive; that being a 10 TB Western digital. We have a clone of that drive. From there I offload episodic events to LaCie rugged thunderbolt drives for myself and another editor and we do our work from there. We then go back to the WD drive just to get a new episode and update the events. Seems to work so far. I am saving many XMLs throughout these stages to insure I have those backups. In the end we'll likely delete the card originals when the series is done.





© CreativeCOW.net