Metadata, P2CMS, P2Log Pro, LogAndTransfer, Raylight and You. A journey into the unseen.
I am venturing into the world of metadata in order to maximize information and efficiency on set and back at the studio. I need to talk this out with someone that knows all of these programs to make sure I am understanding all of this correctly and perhaps offer some suggestions as to what I need to buy. Or perhaps what we really need to do is yell at Apple (in a very tender and gentle voice) and tell them to just support MXF metadata natively. Anyway, here goes:
-It handles AVC-Intra (as soon as 1.2 gets posted)
-Verifies copies (don't need to make disk images with OSX)
-Color codes clips to show you they have been ingested.
-No saving or altering of the MXF files, it simply tracks all of that data and saves it to the cards.
-Can move the MXF files to another machine for later editing back at the studio without much rigormorole.
-No offense Panasonic, thanks for free, but that interface leaves a bit to be desired. I cannot rename multiple clips at once (for instance if I have 40 clips of the same interview, all i want to do with a click or two is name those clips JoeBlow_01 - JoeBlow_40). Or if I have multiple takes of a scene, I can't just rename those in one fell swoop Scene101_Take1-Scene101_Take10. All the clicking will make mince meat of my wrist within hours.
-The manual is tough to translate sometimes, even from the English.
-No support in case something goes wrong.
-I need raylight in order to show the UserClipName Metadata in FCP (not a shortcoming of P2CMS, just of the workflow in general)
-Quick, easy, cheap.
-Gives FCP what it really should have natively.
-No rewrapping of Quicktimes
-Moving project form the shoot drive to the edit drive will be problematic. You would have to have raylight on both the field computer and the edit computer which seems kind of pointless to me or perhaps I am thinking about this all wrong.
-No AVCIntra support (?)
P2Log Pro Pros:
-Robust logging interface. Can rename multiple clips at once.
-Will rewrap the MXFs for you
-XML interface for FCP to send the metadata to FCP
-Lots of access to all the metadata in a concise package
-Still need to rewrap the MXF files
-If you don't want to rewrap, you still need Raylight. Fine, but moving project around is not easy.
-Most expensive (but not cost prohibitive)
-AVC Intra support or lack there of (?)
Log and transfer Pros:
-Interfaces directly with FCP
-no other software needed besides drivers
-All p2 formats supported
-FCP project tracks data so moving projects to different computers is a snap
Log and transfer cons:
-Clunky naming interface (again, one clip at a time)
-Have to rewrap to QT so it doubles the media size requirements
-Can't access pre-named metadata, have to remake it to fit into FCPs database
OK, there's the short of the long of it. So, what am I missing?
What do you guys do and how does it help you? What would you buy?
Thanks in advance.
I think one of the most powerful parts of the workflow is the metadata upload feature which you can do from either the P2CMS P2 Viewer or the P2 Log programs. This would give you the ability to come back in from the field with the clips already named like you mention above.
Then by using the Raylight as the interface for ingest and transfer, it can be a very powerful tool in sorting your clips based on User Clip Names. The way they worked the Iditarod this year is that they named the metadata files after each of the checkpoints. Raylight ingesed from 10 different cameras and cards clips that were recorded with the metadata files and it sorted clips into folders based on the metadata.
I think that initially it seems odd to have to use the Raylight but the fact that it maintains the metadata is the most intereting part because that is the power behind the file based workflow.
I am in the process of writing a paper describing how to go about this, and am willing to share it, trust me it is not finished, just contact me offline and I will send you a rough.
crittendenj at us.panasonic.com
Jan Crittenden Livingston
Product Manager, HPX500, HVX200, DVX100
Panasonic Broadcast & TV Systems
[Jan Crittenden Livingston] "I think one of the most powerful parts of the workflow is the metadata upload feature which you can do from either the P2CMS P2 Viewer or the P2 Log programs."
Right, I forgot to mention that. While this seems like THE way to do it, I just can't trust that every shooter I work with will know what to do with it and that we will be able to foresee exactly what we will be shooting. For instance, if we are shooting Broll for a week, there's no way we will be able to predict what we are going to get. So in that case, renaming after we shoot is the way to go. I wish every shoot was organized by checkpoint. That'd be sweet. Is there software for that? ;)
[Jan Crittenden Livingston] "I think that initially it seems odd to have to use the Raylight but the fact that it maintains the metadata is the most intereting part because that is the power behind the file based workflow. "
Undoubtedly. That's what I am trying to harness.
[Jan Crittenden Livingston] "just contact me offline and I will send you a rough. "
Email has been sent. Thanks for your response.
[Jeremy Garchow] "For instance, if we are shooting Broll for a week, there's no way we will be able to predict what we are going to get."
In a best case, you'd have some PDA-thingy that'd let you edit a text file on a SD card. Wish I had one. (we use a laptop when we're not out in the boonies) Barring that, you could pre-define broll categories. Shooter selects metadata based on what they're pointing at. Then rename (if needed) clips once they're imported as QT refs in FCP.
[Bob Woodhead] "Barring that, you could pre-define broll categories. Shooter selects metadata based on what they're pointing at. Then rename (if needed) clips once they're imported as QT refs in FCP."
Yeah, I could see how that would be cool. I just don't see it happening in reality, but maybe that's just me and haven't put it in to practice yet. What we need is a wireless text shooter from the data wrangler to the camera. Logging while shooting.
I think the most significant aspect not realized in your outline is that working with clips on different edit stations is really very easy with Raylight. There's an option in Raylight to save the newly created QT reference file in the same location as the MXF. So my workflow is:
- transfer clips (either off camera or off cards) via P2CMS onto a 2 drive RAID 1 array. (Mirrored drives for safety.)
- Run Raylight on those files
- (if desired) copy everything to an edit RAID (raid 5)
- do a simple Finder search for .mov files on the folders. the search results (a QT file for every clip) can be dropped into FCP.
- storing the QT ref files with the MXFs means that you don't need Raylight after the initial processing (and also eliminates the possibility of changing options in Raylight later, and thus potentially mucking up your FCP references)
So Raylight can be a "run once, use many" operation.
No help here on AVCIntra.... 500 owner.
"Constituo, ergo sum"
Bob Woodhead / Atlanta
[Bob Woodhead] "There's an option in Raylight to save the newly created QT reference file in the same location as the MXF"
Thank you for your response.
The way I understand it, though is that if you move the MXF to another raid with a different name and folder structure, that relationship is now gone and has to be rescanned which means you have to have a copy of Raylight on every potential workstation. I am sure that would make dvfilm.com very happy and for that I don't blame them, but I am just talking practicality here. Am I wrong about that? From the raylight mac help page:
Locations other than P2 Cards or volumes containing P2 Card images, for example a P2 folder on your desktop, may be scanned by adding them to the "Other Scan Locations" list.
If you select "Store Links in Same Location as Source Files" then the links will be stored in the same folder as the video MXF files. This can make it easier to organize the links, however, the MXF files must be rescanned if they are copied to another folder or drive.
The links created by Raylight may be copied or moved to any other part of your computer, including other disk drives, and they will function normally.
If the MXF files are copied or moved anywhere, however, then you must remake the links by clicking "Scan Now." Use the "Overwrite Existing Link Files" option if you need to replace existing links with new ones. In some cases an old link will still function but there is a delay of several seconds while the system searches for the MXF files. If this happens in the Quicktime Player, you can save the link using File->Save and the link will be updated. If it happens in FCP, open the link with QT Player, then File->Save the link."
And don't you find P2CMS kind of weird? Do you ingest in the database or just export with verify?
Right now I don't think that there is a workflow that will do everything you want. That's why you're posting here, right?:-) You need to be able to have someone on set with a handheld/laptop/iPhone connected directly to the camera, and rename and log clips within the camera. i think this will actually happen, its jsut a matter of when.
In the meantime, what do we do? It depends on your workflow that you need. Its rare that there is a solution that covers every situation. Personally I dont' like the idea of editing my shows with reference files, its one more thing that can get mucked up. I haven't done any testing as to relinking files that get archived and refs that get deleted/moved, but right now I have more disk space than time, so I log and transfer. I use a bare drive in my weibetech enclosure and archive the raw P2 data, put it on a shelf. FCP was made to work with quicktime movies. Not MXF files. until Apple opens the platform to recognize other file types, there will not be a reliable way to get metadata from the P2 card into FCP. For some reason, QT's don't jive with P2 metadata.
Anyway, enough of my rant. As an editor, I despise logging and digitizing. I've always hated it. I can log and transfer, its faster than realtime. I like that. Its one step forward without using some fancy footwork of a workaround that has a greater chance of failing (computer error or human error) and sticking me near my deadline.
These are my OPINIONS, this is the way I work. I know everyone works differently. Most people don't mid using raylight, and it is a cool idea, and it seems to work. But not for me.
Until log and transfer recognizes the metadata within the mxf files, I don't think there will be an easy way of doing this.
Its more fun to ride a slow motorcycle fast than a fast motorcycle slow...
Sorry for the rant. To summarize, yell at Apple.
Its more fun to ride a slow motorcycle fast than a fast motorcycle slow...
Had to come up with a feasible workflow like this for a big, fast turn around, multi P2 Cam shoot in New Zealand earlier this year.
After much testing the best workflow I could find involved;
Using P2 Log Pro to create metadata to load via SD Card to each of 7 Cameras (HVX's and HPX500's).
Metadata included Camera Op name, Description of what they were shooting that day but most importantly a UserClipName that ID'd each camera and created incrementing UserClipNames in camera, ie A CAM TUES 001, A CAM TUES 002...
I would do this for all the cameras before the beginning of each days shoot.
( I think there's also now a free Panasonic tool for creating and uploading metadata to camera so don't go out and buy P2 Log cos it's day is done! )
Next step were the Logging and Transferring stations ( PowerBook G4's ) where full P2 Cards from the field were copied to the edit machine. For this we used P2CMS - this is the one thing it does well and relatively simply, a verified copy of the Contents of each card.
( It does a whole lot of other stuff involving a database or something but forget about all that, it's confusing as hell.)
Raylight for Mac was installed on the edit machine, you only need it on one machine.
This was setup to scan the edit drive and create Quicktime Ref files for any new MXF files that arrived. It was also set to create new names for these QT Ref files based on the UserClipName from the metadata and the actual MXF file name, this is great, in a matter of seconds Raylight will generate QT Ref clips with meaningful names that are ready to edit in FCP and that you can track back to original material if necessary.
I set it to make these QT ref clips in the same folder as the original MXF Video files so each card still exists as a self contained Folder that could be moved around and still be Imported into FCP using Log and Transfer or can be edited immediately using the Raylight QT Refs.
We actually had another step here where we would open the Raylight QT Refs up in FCP on another networked Logging machine. Here a Logger would make logging notes to help the editor and then save a Batch List back to the CONTENTS Folder of the card in question.
At this point we would use Finder to Label the Folder Green for Go and all the editor had to do was open each Batch List in each Green Folder and do a quick reconnect to the Raylight QT Refs and away he went. Hundreds of Logged, HD Clips with meaningful names in FCP.
Say you transferred a Folder representing a complete P2 Card from one drive to another, the clips would appear as offline in FCP but all you have to do is reconnect to one Raylight QT ref file in the new location and FCP will connect all the files in that folder.
It's just if you started moving the QT ref files in relation to the MXF files that they are referencing that you would have real problems.
It worked pretty damn well. Occassionally the Raylight clips would open in FCP sans audio. When this happened it was a simple matter of deleting the QT Refs for that card and on it's next automatic scan Raylight would recreate them and second time round the audio came thru in FCP. ( I think this was due to the network traffic and activity on the drive rather than a Raylight flaw. )
At the end of the job everything was relocated to San Fran for a seperate edit and they did have trouble reconnecting all the QT Ref clips in FCP - In New Zealand we were doing Daily edits with no problem but in San Fran they were dealing with all the media from 7 days so i think it was just too much Quicktime Referencing going on!
I think the answer to my prayers might be just around the corner though...This looks like the business, superceding Raylight, especially if it integrates the camera metedata into FCP the way FCP should!!!
Wow. Thanks a ton for posting this, Josh. Much appreciated.
Unfortunately I haven't had time to do much more research and testing.
I was alerted to this new product as well, but it doesn't help with UserClipName within FCP but it allows direct editing of MXFs right in FCP among other features, without the need for reference movies:
Once I get through this next shoot I am hoping to get into preset metadata, but I know this won't be a reliable workflow as I won't be able to teach it to everyone and I personally won't be on every shoot either.
Raylight seems like a stop gap to me so I can't commit to it right now.
Interesting to hear everyone's variations on the metadata workflow.
I have setup the P2 workflow for the Iditarod since they went P2. This year they used metadata and edited mxf with FCP.
We used P2 CMS to create the metadata files. We identified the 24 checkpoints in the Scenario field, and saved 24 unique files to each shooters sd cards. In the mac finder, we named MTDT01.P2 as something more logical, like mcgrath.p2 and loaded these in to the camera.
We archived the cards each day with HD Log Pro. This allowed for consectutive card offloading and multiple destinations.
One the cards were saved, P2 CMS was used to clean up any metadata fields, like when a shooter missed a metadata change.
In P2 CMS you can change metadata for every clip or selected clips at the same time.
Once the metadata was correct, we ran Raylight on the files.
Raylight has a cool feature where it will make bins based on the Program Name (Scenario).
This created bins for all the checkpoints automatically. Saved a lot of time in the edit!
The Raylight folder was copied from Users>Shared to the drive with the mxf files. Then the raylight folder was imported into fcp.
No need to have raylight on the other systems as long as the initial path doesn't change.
I do find it handy though to have multiple licenses, in case I need to tweak the metadata and re-scan.
Hope this helps,