When to start a new catalog...
We have an arts series with about four stories per episode and 15 episodes per season. We've been putting everything for the season into one catalog, but it's now got 6,000 clips and it's getting slow to load. We wonder what others do - putting every story into it's own catalog would solve that, but that seems goofy too. We can't put an episode per catalog, because we don't know what stories are going to end up in each episode!
Should we use one catalog per story?
Should we instead try to speed things up by reducing the frequency of thumbnails for the filmstrip view?
Are you using one of the server products?
We do and we create catalogs based on acquisition, which is to say, every day we go out shooting the footage is logged into a separate catalog. Same with out of house media as it enters the door.
When we are ready to put together a project, we select clips from these individual catalogs and make a "working" catalog which has the media from all of these catalogs. We usually save the project catalog with the archived project (not in our database, though we probably could).
Naming practices and consistent folder structure becomes crucial in working with this method, at the media tends to scatter quickly across the system otherwise.
8317 Front Beach Rd, Suite 23
Panama City Beach, Fl
phone 850-234-2773 ext. 128
skype username - vidmarv
Yes, we are using the server, with the worker node.
Your response is VERY helpful!
Time to rethink what we are doing.....
great advice! We're finally getting our CatDV system set up here over the next few weeks and this is exactly what we are trying to figure out too.
Walter Biscardi, Jr.
Editor, Colorist, Director, Writer, Consultant, Author, Chef.
HD Post and Production
Biscardi Creative Media
"Foul Water, Fiery Serpent" featuring Sigourney Weaver coming soon.
Blog Twitter Facebook
Thanks, Kevin and Martin.
Here is the twist: (Sorry I don't know all this stuff first hand.)
We use the Worker Node to automatically ingest content into CatDV when it is dropped into a "media" folder on our server. Worker Node pulls those few fields of meta-data available from the XDCam transfer tool, dumps the data into the CatDV Server database, and automatically generates the proxy. All this is very cool.
Thing is, Worker Node is set to deposit new content into a specific catalog. We don't see a way to use Worker Node for the ease of ingest and proxy generation, while ending up with each "story" in a unique catalog.
You can insert a variable in the Catalog name to make unique catalog names. Not all the variables will work but hit the help button in the Worker for a quick list of variables. Examples are to create a catalog name based on a timestamp or a parent folder name. Be sure to have the latest version as there is some nifty new enhancements!
With the Worker, if you don't specify a particular catalog you can get it to automatically create a new catalog with each day's ingests.
Another common scenario is to have the Worker initially put all new clips in an 'In Box' catalog, but then manually drag clips out of that into a more specific catalog as part of the logging process (perhaps one per episode and one for general assets, or one per logger as a way of assigning work to them, or whatever else makes sense in your particular workflow).
Also, as Kevin and Bryson point out, catalogs are just *one* way to browse and navigate to clips. It's exactly analogous to files in a directory: every file has live in one directory (every clip belows to a catalog), but you wouldn't put 100,000 files in one directory (or 100,000 clips in one catalog). The beauty of CatDV is that once you've tagged your clips with metadata you can browse the data just as effectively using smart folders or automatic grouping terms as you can by catalog, so you can very quickly "slice and dice" the view of your assets as required.
the server is the correct way to go with so many clips. It is important to remember that the catalog is just a virtual folder if you like. When you query the Server it will search across all catalogs and produce a "View" This is a set of results that match the search criteria. So if you have a field called EPISODE NUMBER and you search that field in your query it will return all the "hits" for say EP1. This could be EP1 for many productions but it will give you them all. It will be very quick because by choosing to query on that field you are using the power of an SQL databases indexing features. Now this may not be what you want so you have a couple of options. You could now use the interfaces GROUPING option. Click on the Group icon on the top toolbar. You can then select another FILTER criteria. One of these fields could be PRODUCTION NAME. The drop down will show you all of the PRODUCTION NAMES it knows about. Click on the PRODUCTION NAME you are interested in and it will FILTER the view to show you only those clips for that PRODUCTION . SO the filter is EPISODE > EPISODE NAME.
You could of course have specified this in the query to begin with. In the server version click on the magnifying glass icon.. the Query panel appears . There are two views simple and advanced. Advanced has all of the funky geek boolean stuff.. SImple allows you to match you Query window to the Logging Schema . So if you have created drop down lists in the Details panel they are mirrored here.. So from the drop downs without typing you can select your search criteria.. So it could be YEAR > SEASON > PRODUCTION NAME > EPISODE NO
or whatever your original Schema provides for .
Why am I explaining all of this . Well you can see that as long as a clip has metadata associated with it you do not need a catalog to hold it. ( pause for effect )
I tend to think of catalogs as collections of related clips , so they can provide context around the results of a search.. ie CATALOG could contain a days shoot .. if you want to see what was shot before or after a particular clip then ask the QUERY to return the catalog that the clip is contained in.. Its best to experiment. So a catalog is analogous to your folders containing your media and I see a lot of people filing their clips this way. A named folder containing media. CatDv will pick up this information and write it into its BIN field so you could Query the BIN field for name of the folder. ie MY-Shoot-June-10-2010 or whatever.
Its best you start to understand how the DATABASE works so you can simplify your workflows bearing in mind CATDV will "harvest" a lot of metadata when you bring it in.. Check out the OTHER PANEL to see how much is gotten for "free" This is format dependant . If you type stuff into your P2 camera for instance we get that and write it into the tables.
The most important tip I can give you is to be efficient you have to think GOOGLE... This is the best model for handling billions of bits of data . Once you have a tricky Query defined you can save it as a SMART FOLDER ... ie PRODUCTION NAME = CSI MIAMI EPISODE NUMBER = EP1 and call it CSI-Miami-EP1 . Clicking on this in the tree will get you all of the clips for that productions episode one. The upside of this approach is you do not have to move media and it auto updates its results as more clips come in and our tagged..
Thats a lot to take in for the moment ..
The concept that Kevin just dropped on you is the biggest thing in CatDV, it's the moment when you "get over" the obsession with catalogs.
I take partial blame for that obsession. In demo's we tend to work with small amounts of footage and so opening a catalog is the obvious thing to do to show clips. Running a query with a small amount of footage quickly gathered isn't "sexy". Also, in the standalone, you always start from that so it's ingrained. In the field, it's great to have a container, but once a clip belongs to more than one container, i.e. used in multiple shows for example, or that container gets too big, there are problems. What Duggan shows above is usually a 15 second item that goes by in a flash, without being noticed really.
As you said, the minute you have catalogs that are per tape or per episode, things get hard to manage in some people's mind. Kevin has laid out the solution.
I also find that people who are trained mostly to search from the start have less trouble "getting" this. If you start in a shop that has several thousand catalogs you really don't try to scroll down through the browse window to catalog #3789. Whereas if you are the guy starting the system, at first you simply open catalog 5 or 50 to look at clips as it slowly grows.
Naming is a huge deal as you start to do this, as EP1 will be returned in general searches for EP1, EP101 and EP1001 so think ahead. (lets hope you all get a 1000 episode order for a show!) That said, if you use the grouping or tree to browse and "query" that isn't an issue. Also, if you do make a naming mistake, you can easily "bulk edit" things to change them after the fact. This is also eased by Kevin's "narrow it down" method of stepping into a catalog set.
To paraphrase Kevin's tape quote, "The catalog is dead, long live the catalog".
But once again, CatDV proves to be many products in one, with many ways to manage and view. It's all about how you use it.
bryson "at" hidefcowboy.com
The other thing that I'm thinking in reading though these responses (and looking back at the original post) is the realization that you don't have to publish EVERY catalog to the database. More and more, we do our queries and collect individual media in "project" catalog. All of the media that is in this "project" catalog exist in the database, it's just consolidated into one catalog in order for us to generate a FCP XML file to import into the project. While we usually save this "project" catalog (as a file with the archived FCP project), we don't usually publish it to the CatDV database, as it relates only the project that we're currently working on.
As has been mentioned, don't get hung up on the catalogs. Obsess with getting ALL of you media entered into the database (of course using catalogs), then use new catalogs as tools to create individual projects (i.e. offline edit) independent of the database. While I suppose you could publish them, our feeling is that it would render erroneous queries and become confusing quickly. Hope this makes sense.
8317 Front Beach Rd, Suite 23
Panama City Beach, Fl
phone 850-234-2773 ext. 128
skype username - vidmarv