Creative COW SIGN IN :: SPONSORS :: ADVERTISING :: ABOUT US :: CONTACT US :: FAQ
Creative COW's LinkedIn GroupCreative COW's Facebook PageCreative COW on TwitterCreative COW's Google+ PageCreative COW on YouTube
FORUMS:listlist (w/ descriptions)archivetagssearchhall of famerecent posts

Folder Structure and File Naming

COW Forums : CatDV

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Share on Facebook
Scott GoddardFolder Structure and File Naming
by on Apr 23, 2012 at 6:04:50 pm

The more I read up on CatDV the more important I realise the folder structure and file naming is in terms of organising the 'backend' of CatDV. Currently I am using CatDV Professional (non server, non worker node, but certainly coming to that later this year).

I am organising footage from a wide range of sources and a wide range of time dating back many years. It is non project specific but stored in an archive style ready to be used when needed. Lets think of it as a stock footage library as that what it and I imagine a lot of small / medium production facilities could be using CatDV as and what closely represents my own uses.

I am trying to come up with a folder and file naming convention that will best organise the footage in a raw file based non CatDV reliant structure (mainly as I see it to be important as a backup to be able to manually locate incase there is a software issue, loss of computer etc etc) as well as to allow CatDV to 'mine' data from the folders and filenames. From what I have read in a few posts this is something CatDV is capable of doing and becomes very powerful when used eventually with the Worker Node. In fact from what I understand the worker node can in essence be programmed to do most of this organisation for me.

Essentially though we still need to have a structure in place to begin with. Here's what I am currently thinking in terms of folder structure:

RAID VOLUME / YEAR / MONTH / COUNTRY / LOCATION / FILE FOLDER / File (file name will include as much data to make it usefully identifiable)

So something like as an actual example:

RAID VOLUME / 2012 / April / France / Paris / 2012_04_23_france_paris_city_broll / 2012_04_23_france_paris_city_broll.mov

So I can see from this file the date shot and a simple description of what it was. Of course the CatDV metadata will hold much more information on it, such as cameraperson, producer, markers with shot details, weather etc etc

I am wondering what others use as a folder structure and naming convention? Of course it will be different for everyone but perhaps there is some general rule to doing this kind of structure. I would be interested to know how some of the bigger facilities organise their data especially when dealing with tapeless clip based media (hence the importance of the file folder name).

Any advice on this would be greatly appreciated especially in reference to a long term structure that can adapt to the worker node and server solutions.


Return to posts index
Reply   Like  

Robb HarrissRe: Folder Structure and File Naming
by on Apr 23, 2012 at 6:49:59 pm

Well from my perspective you're over thinking it.

I'm in pretty much the same situation. about 16 years of footage, dating from the time we switched from film to video. For the last 13 or so years we've created a great deal of derivative product from all the different material we've shot and continue to shoot. It's all documentary in style. Not a spoken word is scripted. Lots of interviews and lots of transcripts. We create video-based products that are sold all over the world.

A few years ago we even changed the way our accounting works to match the way we actually create product. Whenever we shoot it's research towards making an at-the-time-undetermined project. Product creation doesn't begin until we sit down to put together a project in edit. The best way for us to work this way, and keep footage somehow compartmentalized and still make some sense of it all, was to separate it out by shooting location. So we have this factory, or that office site, or that hospital or school. Reel names (we shoot HDCam on a 750) use as two or three letter designator for the shooting location. So I can see a note or a line out of an EDL, or a tape case and know immediately what "shoot" it came from.

Ok. So the majority of the time I'm using FCP to load the footage. The project goes under the location name, say "Unity Hospital." What become important later is the path to the directory Unity Hospital. Doesn't matter how or where that footage goes later. I just need to find it. I use FCP to mark sections to load and label them such as: UN07 cafeteria cover 1.mov or UN10 Lacy Darcy interview part 2.mov {Note that I'm using ProRes for my files). When the shots are captured I bring them into CatDV, which will remember just exactly where the footage is. I can then parse them out even further with all sorts of key identifiers and markers and transcripts. I also create proxy movies of everything that live on a separate drive attached to one of my systems. Because I'm using "paths," the structure is the same as the original footage.

Lately I've been moving footage off onto external drives for storage, and to clean out my systems. I'm sure some lucky souls get to use CacheA but for now we're using raw external drives. It's quick and easy and I can mount a drive using an eSata dock directly to one of the systems and edit from the raw drive when I need to. But the real change has been moving the original footage off to one of my archive drives FIRST and then bringing the footage into CatDV and making proxies from there. I'm doing the vast majority of my editing using the proxy movies, NOT the originals. As a result I have 100 of my footage from ALL shoots available ALL the time. Originals don't come into play until I'm working on finishing. Then I can mount archive drives and conform/consolidate/relink or do whatever. It's been quite slick. And by moving the originals to an archive drive first I'm not reconfiguring CatDV all the time to find the footage for both the originals and the proxies. I also have a portable clone of the proxy drive for working offsite (I used to just call that home). Because we're using "paths" for proxies CatDV (and Final Cut) always thinks it knows where to look for a shot.

Sure, you can do what you suggest, but I think you're adding a layer of complexity that's not really going to help you that much and it will eat up even more prep time. Takes me back to the days when I setup the table of accounts for my old company's accounting system in Quickbooks. I had everything parsed out to the Nth degree. I could have told you how much black gaffer's tape was used on training projects versus commercials on any named Thursday. Of course I never did, nor did I ever actually need that level of detail. In the end it was a big lesson learned. One of the big lessons with CatDV is that you can add fields and descriptions later without messing up everything you already have. Are you willing to go back and redo your directory structure if someone suddenly decides if they want to know if shots were done under cloudy or sunny skies? Ouch.

Non-linear: all the time and nothing but.


Return to posts index
Reply   Like  

Scott GoddardRe: Folder Structure and File Naming
by on Apr 23, 2012 at 7:19:28 pm

From what I understand you need to have a path that is constant for the 'Full Res' footage. The folder and file structure is then identical for the proxies. Are you able to have multiple drives for the full res that you can slot in and out? How does this affect your proxy creation?

I will not be keeping much of the full res on file to start with, currently bluray data and then cacheA. The proxies are kept on a Raid 5.

I have read conflicting arguments for having a deep folder structure and for not (like yours). Metadata like weather (cloudy/sunny) or course will not enter the folder structure or file name. I don't really see it as being too complex, almost essential.

Lets say CatDV (for what ever, hypothetically) stopped working, how well would your structure hold up?

In regards to having everything in triplicate, could you expand on this? The reason I may need them like structured like this is for using the worker node to organise my footage in the future where essentially it can organise footage into the right year and country and location based on given fields in CatDV.


Return to posts index
Reply   Like  


Robb HarrissRe: Folder Structure and File Naming
by on Apr 23, 2012 at 7:38:43 pm

Ok, there are some semantics that we have to be careful of here. It's EASIER if your paths are consistent. That MEANS: if you put them in one place, it's hard for CatDV to find them if you move them.

Translation: Don't keep changing where you store them. It doesn't mean you can't store them in multiple places. CatDV keeps track of where everything is. If you move them it doesn't know where to look. However, if you move them you CAN tell CatDV where to look and all will be fine.

So, best thing to do is put them in a place, read them into CatDV and then let CatDV make the proxies. But because you are using paths (and not tape names) you can use other software, like Compressor, MpegStream Clip or something else to make the proxies. You just store them on your proxy drive(s) using the same structure: Volumes/Archive 1/Paris becomes for the proxies: Proxy Drive A/Volumes/Archive 1/Paris

you don't want to have too deep a directory structure because you'll run out of space for the OS to deal with it. I'd have to look up what the limits are (someone chime in here if you know), plus it gets very complicated to manage. You keep it simple and let CatDV do the heavy lifting. When you go to the workgroup version you'll see how easy life becomes because you don't have to deal with all these separate catalogs. All becomes as one (grasshopper). You just to a query and CatDV finds all the shots/clips for you. I do a LOT of drag and drop from CatDV to FCP directly. Works wonders and cuts down hugely on edit prep. Everything is already loaded and ready to go, as proxies.

Holding up? Each file name says what it is. The shooting location is already defined. After than I need the specifics of the shot itself and the other information is pretty much in the way. Again, CatDV does the heavy lifting.

And in triplicate: well your file name already has the date/country/city/broll and more, and yet you have directories that repeat the information. Actually if they're all in the same directory a simple directory sort will give you all you need. CatDV isn't going to die or blow up or something. Or at least it's not likely. I do keep copies of catalogs locally in a DropBox folder and those are backed up six ways to Sunday. My date fields INSIDE of CatDV are what are important. If I tried to put the same degree of information in the directory structure and then duplicate it in the file name, well, talk about a plate of spaghetti. Start simple. You can always get more complicated later (inside of CatDV).

Look at what Bryson put up about how NASCAR uses the system. You're not going to see that kind of directory structure.

Non-linear: all the time and nothing but.


Return to posts index
Reply   Like  

Robb HarrissRe: Folder Structure and File Naming
by on Apr 23, 2012 at 7:47:22 pm

Here's what the directory structure looks like on one of my proxy drives.

I do like having the reel name as the prefix to every file name. Oh, and I window burn all the proxies and put the reel name on all of them. That works great when they're being referenced by someone who isn't used to dealing with footage.



Non-linear: all the time and nothing but.


Return to posts index
Reply   Like  

Scott GoddardRe: Folder Structure and File Naming
by on Apr 23, 2012 at 8:19:33 pm

I see, so your 'Archive' folder relates to a drive?


Return to posts index
Reply   Like  


Robb HarrissRe: Folder Structure and File Naming
by on Apr 23, 2012 at 8:23:20 pm

Yup, each "Archive XX" is a raw drive. Mostly they sit in a fireproof safe. I still have the tapes but once they're on the raw drives I don't actually have to capture them again. That saves a bunch of time downstream. It costs no extra time in prep or outside of traditional workflows.

Non-linear: all the time and nothing but.


Return to posts index
Reply   Like  

Scott GoddardRe: Folder Structure and File Naming
by on Apr 23, 2012 at 8:15:19 pm

Thanks for that. Yes I am certainly all for CatDV doing the heavy lifting. Yes the server solution is where I am heading in terms of multiple catalogues. This I am fully behind. Have you had any luck with getting CatDV to take certain elements from the file name and use that to populate fields or vice versa? Perhaps its only possible with the worker node.

Bryson referenced Nascar here:







But did not go into folder or file name specifics, did you see something else from him? I searched but could not find it in the forum.


Return to posts index
Reply   Like  

Robb HarrissRe: Folder Structure and File Naming
by on Apr 23, 2012 at 8:21:38 pm

Bryson didn't go into that level of detail. But I believe that his description of their workflow details how they put your level (and mine too) of detail into CatDV rather than trying to create it in either directory structure or file names. I mean after all, that's what CatDV is there for. If you're going to do all the work, why have it.

I don't have the Worker node. I create my first level of organization when I capture the footage using FCP 7. So far we rarely see file-based footage, though I'd like to get there. And I like long pieces of footage filled will shots and interview. The worst thing in the world for productivity is thousands of individual shots. I can't imagine what it would be like if an interview came in a word at a time.

Non-linear: all the time and nothing but.


Return to posts index
Reply   Like  


Scott GoddardRe: Folder Structure and File Naming
by on Apr 23, 2012 at 8:38:16 pm

Yes tape based was/is easier. FCP7 was not made (not much software is) for editing 'clip' based footage. I can't stand it. Its incredibly unproductive to be sitting there double clicking footage to watch a 10 second clip.

Mostly when I am editing XDCAM EX I end up just making a timecode accurate play out of the entire cards clips. So basically i get one long say 50 minute clip (as you would get after ingesting a HDCAM tape.) I keep this native as XDCAM EX with the audio exactly the same as separate tracks. This really does not take long to do in FCP and makes editing as quick as you are currently doing it (as if it was off a tape). Of course it is completely backwards when you think about it.

Ideally Sonys XDCAM transfer would have an option to 'link all clips' with markers for shot change and any other metadata change that was previously recorded. I simply do not know how most editors work with a traditional tool like FCP7 and clip based footage, especially if they are working to tight turnarounds as I often do.

CatDV deals with xdcam very well with the MXF add on if you are keeping it all in its BPAV folder (as you should do).

Its just a shame that clip based is not really how our brains work or edit software works. I know that after I shoot for a day I have a vision of that days shoot as one long clip, so does the producer and the client. Thats how I remember it, thats how the FCP viewer deals with it and how we would all like to edit with it. I don't think there is a usable solution yet to this but I may be wrong.


Return to posts index
Reply   Like  

Robb HarrissRe: Folder Structure and File Naming
by on Apr 23, 2012 at 8:48:33 pm

I convert everything to a unified format anyway (ProRes). CatDV has an ability to combine clips, though I haven't (yet) had occasion to use it. I know there are ways of stringing clips together from XD cam. Would make a lot of sense for what you're doing, as part of the prep process.

Non-linear: all the time and nothing but.


Return to posts index
Reply   Like  

Scott GoddardRe: Folder Structure and File Naming
by on Apr 23, 2012 at 8:56:20 pm

Yes it makes sense to do it as part of the edit prep for sure. Its just that keeping it as a BPAV folder for archive purposes is essential (too much to go into here).

I don't convert to prores at that stage because of time constraints, mostly its not graded much so there is not much need when speed is of the essence. You only loose speed with xdcam if you are doing lots of effects in FCP. Going further off topic I just built a system to run Premier 5.5 with the mercury engine and a Cuda Nvidia card. 10 streams of HD prores / xdcam all scaled to fit on screen at once, all rotating, all with grading and effects. Playback full quality, full frame rate out to client monitor in real time with NO RENDERING. Unreal. Apple lost another pro..


Return to posts index
Reply   Like  


Robb HarrissRe: Folder Structure and File Naming
by on Apr 23, 2012 at 9:12:28 pm

Well maybe it's something the worker node can do—string together clips and save it as something useful.

I've been looking at Premiere as well, and even going back to Avid after 11 years away from it. I recently dropped Soundtrack Pro and adopted Pro Tools. Quite the learning curve, but what I need is robustness. I cannot afford to suffer all the crashes that STP gave me.

Non-linear: all the time and nothing but.


Return to posts index
Reply   Like  

Scott GoddardRe: Folder Structure and File Naming
by on Apr 23, 2012 at 9:16:34 pm

If its the full pro package you are after check out the new version. I use audition a lot for audio and its great, so much more stable than STP. Check out the new colour grading tool in CS6:

http://blog.vincentlaforet.com/2012/04/15/adobe-production-premium-cs6/


Return to posts index
Reply   Like  

Robb HarrissRe: Folder Structure and File Naming
by on Apr 23, 2012 at 9:31:30 pm

Yeah, I have to look at the new CS6. I'm not in much of a rush. I have a lot invested in FCS 3 systems and AJA cards and the like. Plus I'm really dependent on the integration between CatDV and FC 7. That really means more to me than anything else. That and keeping my HD deck running.

Non-linear: all the time and nothing but.


Return to posts index
Reply   Like  


Robb HarrissRe: Folder Structure and File Naming
by on Apr 23, 2012 at 7:10:12 pm

Looking at your proposed directory structure and file name you have everything in triplicate. Just as it is you could drop four levels of directories.

Non-linear: all the time and nothing but.


Return to posts index
Reply   Like  

Robb HarrissRe: Folder Structure and File Naming
by on Apr 23, 2012 at 7:14:21 pm

Well from my perspective you're over thinking it.

I'm in pretty much the same situation. about 16 years of footage, dating from the time we switched from film to video. For the last 13 or so years we've created a great deal of derivative product from all the different material we've shot and continue to shoot. It's all documentary in style. Not a spoken word is scripted. Lots of interviews and lots of transcripts. We create video-based products that are sold all over the world.

A few years ago we even changed the way our accounting works to match the way we actually create product. Whenever we shoot it's research towards making an at-the-time-undetermined project. Product creation doesn't begin until we sit down to put together a project in edit. The best way for us to work this way, and keep footage somehow compartmentalized and still make some sense of it all, was to separate it out by shooting location. So we have this factory, or that office site, or that hospital or school. Reel names (we shoot HDCam on a 750) use as two or three letter designator for the shooting location. So I can see a note or a line out of an EDL, or a tape case and know immediately what "shoot" it came from.

Ok. So the majority of the time I'm using FCP to load the footage. The project goes under the location name, say "Unity Hospital." What become important later is the path to the directory Unity Hospital. Doesn't matter how or where that footage goes later. I just need to find it. I use FCP to mark sections to load and label them such as: UN07 cafeteria cover 1.mov or UN10 Lacy Darcy interview part 2.mov {Note that I'm using ProRes for my files). When the shots are captured I bring them into CatDV, which will remember just exactly where the footage is. I can then parse them out even further with all sorts of key identifiers and markers and transcripts. I also create proxy movies of everything that live on a separate drive attached to one of my systems. Because I'm using "paths," the structure is the same as the original footage.

Lately I've been moving footage off onto external drives for storage, and to clean out my systems. I'm sure some lucky souls get to use CacheA but for now we're using raw external drives. It's quick and easy and I can mount a drive using an eSata dock directly to one of the systems and edit from the raw drive when I need to. But the real change has been moving the original footage off to one of my archive drives FIRST and then bringing the footage into CatDV and making proxies from there. I'm doing the vast majority of my editing using the proxy movies, NOT the originals. As a result I have 100 of my footage from ALL shoots available ALL the time. Originals don't come into play until I'm working on finishing. Then I can mount archive drives and conform/consolidate/relink or do whatever. It's been quite slick. And by moving the originals to an archive drive first I'm not reconfiguring CatDV all the time to find the footage for both the originals and the proxies. I also have a portable clone of the proxy drive for working offsite (I used to just call that home). Because we're using "paths" for proxies CatDV (and Final Cut) always thinks it knows where to look for a shot.

Sure, you can do what you suggest, but I think you're adding a layer of complexity that's not really going to help you that much and it will eat up even more prep time. Takes me back to the days when I setup the table of accounts for my old company's accounting system in Quickbooks. I had everything parsed out to the Nth degree. I could have told you how much black gaffer's tape was used on training projects versus commercials on any named Thursday. Of course I never did, nor did I ever actually need that level of detail. In the end it was a big lesson learned. One of the big lessons with CatDV is that you can add fields and descriptions later without messing up everything you already have. Are you willing to go back and redo your directory structure if someone suddenly decides if they want to know if shots were done under cloudy or sunny skies? Ouch.

Non-linear: all the time and nothing but.


Return to posts index
Reply   Like  

David EspRe: Folder Structure and File Naming
by on Apr 24, 2012 at 10:51:21 pm

I'm a one-man operation, learning about CatDV, here is my system, followed by a few further questions.

For in-studio work, I have a RAID and corresponding backup drive.

Otherwise I have a number of external drives, each duplicated. NTFS formatted. Drives tend to be client or purpose themed, rather than sequential "Archive" etc disks. Folder structure in each drive is:

_Media
_Library
_Projects
2012-04-24 (Acme) Charity Volunteer Day
010 Admin
020 Source
030 Projects
Adobe CS5.5
Intro 007.prproj
Vegas
WebVid 003 (Veg09).veg
040 Renders
100 Products
Web
DVD

Where:
  • [020 Source] contains subfolders for different scenes/cameras, music, foley etc.
  • Dates are in reverse order and numbers have leading zeroes to allow simple text-based sorting
  • "Acme" would be whatever client/company name. For personal projects the client is "Me".
  • Prefix numbers, there to provide for tidy sorting, jump in 10's etc. to allow for possible future insertions (e.g. at "015"). Yes I was an old BASIC programmer...
  • [_Projects] are greater-projects, as opposed to application-specific [Projects], which are really sub-projects, and for example in the above it may be that [Intro 007.prproj] in Adobe CS5.5 exports an intermediate Cineform file into [040 Renders] that then gets used in Vegas's [WebVid 003 (Veg09).veg].
  • Anything in [040 Renders] can be regenerated, and thus need not be archived. At project completion, the final products get moved into their appropriate subfolder of [100 Products], which is retained. The whole folder structure (possibly minus non-critical renders) is then considered (and recorded on spreadsheet) as archived.

Currently I record project locations on a spreadsheet. Sometimes it is necessary to temporarily store projects on one disk then move them subsequently (eg following acquisition of a new disk). Sometimes the disk letter assigned by Windows can get changed by Windows eg should that letter conflict with that of some other volume like a temporarily plugged-in memory stick or a new NAS that claims multiple letters. So I guess I also need a spreadsheet of disk letter assignments and a disk letter label on each disk.

It would be great if CatDV could track file changes eg movement (or apparent movement due to migration on/off the RAID, drive letter change, path (file and folder) name changes, project or file deletions etc. I suppose that CatDV could in principle track such changes efficiently if CatDV itself were used to make/manage such changes, as opposed to doing them externally via Explorer/Finder or backup software. Or does CatDV already have such media management capability?

Perhaps the Windows version ought to have an option save its own disk-identifying GUID (probably-unique random signature) to each volume (so it doesn't have to rely on disk letters that can be unreliable and only number 26). Mac OS does this inherently of course. Also, why not have an index file/database just for that volume. Then any instance of CatDV (even on someone else's computer) would be able to pick up the results and allow browsing etc straight away. Then there's the cloud...

Or does it / will it do some of these things I have mentioned?


Return to posts index
Reply   Like  


bryson jonesRe: Folder Structure and File Naming
by on Apr 25, 2012 at 10:18:22 pm

Hey guys, just back from my post-nab vacation. (much needed.)

In the end, the more metadata in the file and folder names the better, of course but as has been discussed the most important thing is that there is a structure at all.

Folder and filenames mostly need to be unique, that's important and harder to do as you grow larger. A lot of DAM's actually rename the asset to have a GUID as the name and so you don't even have a record of the filename. (This is done to guarantee unique names.) I hate this but that shows you how seriously some people take the db. In fact, a lot of heavy database folks would shudder at the idea that we use the actual user given file name.

We are not generally in that world. I find that for most shops, a date based system works fine. Year/Month/Day and then go from there.

However, if you don't do a lot of projects and you actually work on a project basis, (let's say more than one a week) as opposed to say news or stock footage, you can pretty much do what you like since you're talking about 50 project folders.

Depending on the size of your operation, you can choose your way. I find that humans are remarkable in their ability to remember things. If it's your data. But... if it needs to be handed off to other folks for the future, do yourself a favor and add some metadata to your filenames and folder names.

Oh, have you seen Focal Point? ;)

bryson

bryson "at" hidefcowboy.com

hidefcowboy.com


Return to posts index
Reply   Like  

Scott GoddardRe: Folder Structure and File Naming
by on Apr 26, 2012 at 5:07:04 am

Welcome back to reality Bryson ;)

Glad to have your input, and its reassuring I am on the right track with the importance o folder and file metadata. I can see big shops needing individual file names, it makes sense but for anyone smaller a simpler system with CatDV doing all the heavy lifting should not run into too many problems. If anything the media folder seems to be a useful way to divide content up, especially when dealing with clip based tapeless media. So date, country, location, two word description etc

Does CatDV have a way of generating a continuos number? for example if I wanted to have a field for a unique asset set number (I am thinking of using the Reel field for integration into FCP).

So lets say I have a shoot on XDCAM EX or P2. I could take a cards worth of data and give it a number 1088 in the reel field (or similar user created one). Is there a way that when I come to do the next cards worth of clips I can go to that metadata filed and CatDV will auto prompt the next number in the sequence? I can probably see this manually using the drop down I was just wondering if it has an intelligent way of controlling those numbers?

Scott

Néo Vérité Limited
http://www.neoverite.com


Return to posts index
Reply   Like  

Robb HarrissRe: Folder Structure and File Naming
by on Apr 26, 2012 at 5:18:59 am

I always like to have human-readable data. Numbers don't do much for me unless there's text to give it context.

Non-linear: all the time and nothing but.


Return to posts index
Reply   Like  

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Share on Facebook


FORUMSTUTORIALSFEATURESVIDEOSPODCASTSEVENTSSERVICESNEWSLETTERNEWSBLOGS

Creative COW LinkedIn Group Creative COW Facebook Page Creative COW on Twitter
© 2014 CreativeCOW.net All rights are reserved. - Privacy Policy

[Top]