Organizing Media Storage by Project Number
(Pardon me if this has been answered already. My brain is too full to remember.)
We are setting up CatDV, with the worker node, to handle a department that produces about 30 projects a year. It seems that ideally, the media would be stored on the central server in this hierarchy: ProjectNumber/ReelNumber/Clip
This would allow easily tracking how much storage each project is using, eliminate any reel number duplication between projects, and make the archival/deletion process much easier.
Question is, can we somehow use Worker Node to put clips into the correct project/reel storage folder, based on the "creator" metadata within the clip? We plan to put our project number into this field using the Sony transfer tool.
We really don't want to have to set up a new watch folder for every new project.
You could easily do it with a naming convention. It'd take a bit of doing, you'd basically have conditions that would run scripts based on the name and then remove the code from the name to take it back to the original "sony" name. (if desired). Then you could define paths based on those names, but that's a lot of scripting.
There's a simpler way -
One question though, will you be organizing the clips into a folder structure in the "watch folder" or just dumping them in? Since the Sony software is so manual anyway, couldn't you just put it into "ProjectNumber/ReelNumber/Clip" in whatever drive you were exporting/rewrapping to? If you had that, then by setting the watch folder as root, you could have a worker script that picked up the path and recreated it inside your final storage. You just drop the whole group of folders into the watch.
Not sure if that helps, but I can't see as why you'd ever just drop clips into a watch folder randomly anyway unless you have a really unmanageable crew.
I say, save the trouble of putting the info into the metadata and drop the footage into the correct path structure as you rewrap, but I may be missing the point.
Can you speak to how you are using the Sony software? Are you exporting to the final storage or some sort of interim drive, and also, why would your crew not export into a folder structure as they are doing the exports?
If they are creating a folder structure on an interim drive, (such as a firewire drive), then the Worker can just easily mirror that structure as it moves the clips to the final location.
Check out the Worker's "read me" file to get the variables to do that and read the section on defining a new root for the script.
If I'm missing the point, shoot me a little more info please.
Also, if there's a way to look into the metadata and act on that earlier, I'd love to hear about it from the home office but I'm not aware of that other than what's in the conditions panel.
bryson "at" hidefcowboy.com
To answer your question - we are game for whatever is the easiest approach.
When we come in from the field, we immediately load the media (either XDCam disks or EX3 flash cards) onto the Mac desktop and rename the volume with the reel number. (The reel number is incrementing number within the "project").
Then, we open the batch of clips into the Sony tool, and add the project number into the "creator" field and a description. The project number is duplicated for all the clips, and usually the description too, since we don't have time or see value in describing each clip uniquely. We want to add this metadata here (rather than in CatdV), so it gets embedded into the clip.
Then, we make a copy for backup purposes, and move the media into the worker node drop folder.
It's at this stage where ideally the worker node would detect the project number, and make a folder on the server for that project if it doesn't already exist, make a folder for the reel within the project folder, and put all the clips from that reel into it. Same goes for the proxies. With this structure, we can easily delete the whole project from central storage, after we have archived anything we want to save. And we'll never have a naming conflict when two projects use the same reel number in their naming scheme.
I'm not sure I follow your suggestion - perhaps you are saying there is a way to do this?
Anyhow, I hope the explanation above makes it more clear what we are trying to do, and look forward to your further comments.
I think what you propose raises some interesting points. What you are proposing is what I see many times at smaller shops that have shared storage. They have figured out rightly that the key to success is a well understood Folder structure that is enforced across the facility.
So what you want the Worker to do is to automate this to some extent by having a single drop folder and using Worker conditional testing create and place the files in the correct places on the SAN based on the CREATOR field. I have implemented something similar in Televisa in Mexico City where multiple ingest machines place capture to a central folder and based on naming conventions ( as Bryson suggests ) move stuff to the correct folders . This has been operating so smoothly over the last 12 months that I have just returned from building a further two Xsans to accomodate expansion and I did not have to change a single script.
You say you are using the Sony software so I am presuming you are using a LOG and TRANSFER methodology and re-wraping the SONY files to Qtime you reason that this gets embedded in the clip and that this is a good thing. So firstly I would ask you too look at these files in CATDV in the OTHER TAB of the interface which presents all of the metadata that is embedded inside the clips that CATDV can read.. You may be suprised just how much their is . However it should be pointed out that the re-wrapping process may throw a lot of metadata that is present in the source media, which is why I am a fan of native workflows which are possible using MXF4Mac or Calibrated Softwares quicktime extenders. If the Creator field is present you may be able to use this to create scripts in the WORKER to do what you desire . However you further write .. "we don't have time or see value in describing each clip uniquely" This I think is a missed opportunity. We have clients with many many petabytes of data and I have seen that essentially folder based operations for editors/producers days are numbered as Workgroup type workflow practices such as software like CatDv enable become more widespread. In short we actually do not want people going anywhere near the folders on the SAN. A Smart Folder in CATDV can give you precisely the same functionality plus the harvested and the added metadata and due to its filtering mechanisms very fast pre-edit functionality due to the ability to do global tagging and Speed logging . In addition scripts can be triggered from within CATDV that to ftp and transcoding and other sundry day to day tasks that are the moment falling to Editors to do but could be de-skilled by the addition of automations. So I would urge you to think beyond just the folder approach to asset management and how CatDv can move and archive files under its control , which means the database is updated to reflect these changes without intervention as this leads to consistency which is after all your desired outcome. I make these remarks to you and also with a desire to spark off general discussion about these areas of interest within this forum. Those that know my patter will tell you I have a simple dictum .. "Where is it and What is it?" as applied to asset management. You correctly have identified the Where part of the puzzle I want to stimulate discussion further about the "What is it" part. I think I will leave the discussion here for the moment or the post starts becoming a "War and Peace" edition
Hope this helps
Thanks for your lucid and detailed response, Kevin.
I will look more closely - from what I understand, the re-wrapping does not eliminate any meta-data. When we look in the "other" tab, we see a lot of it!