FORUMS: list search recent posts

Catalog Limitation Organization Questions

COW Forums : Square Box CatDV

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Jeff Schaap
Catalog Limitation Organization Questions
on Nov 29, 2010 at 7:55:22 pm

Our organization is just at the beginning of implementing CatDV Server. I am still in the process of understanding how it works and how to properly deploy it for their purposes. All that to say maybe I am missing some big piece of the puzzle here.

Our organization shoots and has shot footage all over the world. This footage gets used in various internal projects as needed. Sometimes footage is shot for a specific purpose or project but often footage is shot without knowing exactly when or how it will be used. I say to give context to my main question and to point out that we don't operate the way that most post houses do.

Initially I thought the best way to organize our existing, file based (P2) footage was to create an elementary folder structure and then use CatDV to add more specific meta-data. I imagined that we would keep all this footage in one huge database (catalog) that would be search-able by all our production staff.

However, I soon discovered that CatDV catalogs have a size limitation. They really can't create or handle catalogs that have more than 3000 records. Thus, we are forced to divide our "master" catalog into multiple catalogs and search "across" those catalogs. At least this was the proposed solution...

The problem for us is WHERE to create those divisions... I originally thought it would be best to divide the our footage into six catalogs, by continent they were shot in. However, some of those sub-folders contain more than 3000 clips each... so no dice. The next "level down," so to speak, would be to divide the catalogs by country where the footage was shot. But that would likely generate 20-40 catalogs... that seems like a nightmare to manage! I fear that with so many catalogs there are going to be a lot of mistakes or misfiling of footage.

So my question is how do people typically handle this issue? Honestly, I am a bit shocked that CatDV can't handle this limitation transparently. Why do I have to create and manage sub-catalogs??

Help. Sigh.

Jeff Schaap


Return to posts index

Kevin Duggan
Re: Catalog Limitation Organization Questions
on Nov 29, 2010 at 9:28:49 pm

Hi Jeff
I have written about these types of "issues" elsewhere within this forum about the Catalog restriction, so do a quick search here. In truth the first thing you should realise is that on CatDv Server all queries are global across all catalogs. A query for "India" will return all records that match this keyword no matter how many records this maybe. However you then have to think if this returns 20,000 hits how useful in practice is this too me. So lets try and break things down a little further . If you look into the prefs you can see that you can restrict the number of records that Catdv loads from its database , to maybe 3000 or maybe 50000 its up to you , do you really want to wait whist Catdv loads 50,000 records into RAM? If a search or query returns this amount of hits then perhaps you should give it a little more specific information. Why not use a date field for instance and search within a COUNTRY field and a DATE field that you may specify and do an indexed search for INDIA and 2008 , this will ignore all values that fall outside of these two criteria and return results for these criteria incredibly quickly and load the metadata and thumbnails from the server into RAM. Then the fun can begin by using GROUPINGS , a drop down list of other available fields to add maybe another field to Filter further what we have already loaded into the CATDV browser. Maybe we can now filter on a field called CAMERAMAN ...Selecting this field in GROUPING you will see a list of every cameraman who shot anything in INDIA in 2008 , by selecting each name in turn you may turn up what you are looking for. I try to explain this as a more human edit centric way of searching, . I have lost track of the number of times you have the conversation with a director or producer and the conversation is along the lines.. " We shot it in India in May with Fred , in Mumbai. So translate that this into how you log and consequently search in CATDV. You then realise then that the catalog is just a convenience much like a folder is on a computer.You can give it a name but it is not a fundemental orginisational tool. The key is the schema or list of fields. CatDv with its query mechanism will present you with a list of these fields and values. So you can easily search on the same criteria with which you logged your footage. Taking care with the selection of these metadata fields is the key to success. I advocate using as few "freeform" fields as possible . Instead use Picklists. Groupings and Multi`s to provide a consisten logging interface. Utilise Global tagging to apply quickly values to all selected clips in the browser and then do a second pass of the clips for discrete and unique data about individual clips. You say you are using P2 , then you are in luck because CATDV can harvest metadata from within these clips if you utilise this ability within your camera. So for instance CAMERAMAN data and memo information can be gathered automatically .
You are probably familiar with the P2 viewer but if you use CatDv and then select all the assets you can copy them to any part of your SAN and CATDV can place all clips from a project within a single CONTENTS folder , try it SELECT all assets then right click to copy to a specified destination . All cards can be placed in one folder.
So in short don`t think catalogs , think metadata tags..

Hope this helps


Kevin Duggan
Catdv



Return to posts index

bryson jones
Re: Catalog Limitation Organization Questions
on Nov 29, 2010 at 9:48:26 pm

Long live the catalog - the catalog is dead.

Reversed a bit, but that's the best way to put it, lol

Once you have a server and are using queries, the catalog structure dissolves just like your folder structure. It's all just returned search results and the "browse" is done by "Grouping" and saved searches in CatDV, if at all.

I use the analogy, of the old days of Yahoo where we browsed by topic category to find a page, until that became useless as the web grew. Now Google makes our "categories" for us and sites are divided as they each choose to be.

bryson

bryson "at" hidefcowboy.com

hidefcowboy.com


Return to posts index


Rolf Howarth
Re: Catalog Limitation Organization Questions
on Nov 29, 2010 at 11:20:56 pm

There's no hard and fast catalog limit, it's just that when you "open" a catalog CatDV loads all the clip metadata (log notes, keywords and thumbnails) into memory. The thumbnails especially are very memory intensive.

How are your files organised on disk? Just as you wouldn't put hundreds of thousands of files in one enormous flat folder hierarchy (see how long it takes the Mac Finder or Windows Explorer to open up a window on that folder if you do!) you wouldn't normally put all your clips in one catalog. Catalogs are just one way of organising your files in CatDV, however, and there is no problem having many catalogs, or searching seamlessly across all of them.

One suggestion would be to create a new catalog for each shoot, or each time you ingest a bunch of files. The name could even be something auto-generated like the current date and time (if you use the Worker Node it automates that part of the process). As long as all the clips are tagged with the continent and country where they're shot (along with other information) you can easily browse by country even though the clips reside in many different catalogs.



Return to posts index

Jeff Schaap
Re: Catalog Limitation Organization Questions
on Nov 30, 2010 at 2:45:10 am

Thanks again for the responses. Yes, moving forward, I can create new catalogs based on the acquisition of footage. If all the catalogs I create can be seamlessly searched as one large database then that is great.

What is a more of a problem right now is dealing with the thousands of files we already have. You asked about our structure. It looks something like this at the root level:

Continent 1
Continent 2
Continent 3
Continent 4
Continent 5
Continent 6

with country sub-folders beneath them that look like this:

Continent 1
|___Country 1
|___Country 2
|___Country 3, etc.

Continent 2
|___Country 1
|___Country 2
|___Country 3, etc.

Continent 3
|___Country 1
|___Country 2
|___Country 3, etc.

and then region folders:

Continent 3
|___Country 1
. |___Region 1
. |___Region 2, etc.

Most of the video clips are in the region folders. Some that cannot be classified reside at the country level.

I suppose the best way to handle it is to create a catalog for each region, since the total number of clips inside those folders is well below 3000, then create country catalogs for those clips not in the region folders. I assume this is the most logical way to do things but if not please point me in the right direction.

My only question is what happens if any of the video clips are moved subsequent to them being imported in these catalogs? If, after having been cataloged, I discover clips the wrong folder and move them to the correct location won't that invalidate the catalog(s)? What do you do to handle that? Can CatDV move files and update the appropriate catalog(s)?

Thanks,
Jeff Schaap


Return to posts index

Kevin Duggan
Re: Catalog Limitation Organization Questions
on Nov 30, 2010 at 8:44:15 am

Hi jeff
just like an NLE CatDv can update its media location via the Media Menu or even attach media , similar to FCP , if you know that something is the correct file , even though some of the attributes do not match. The file paths are then updated and you are good to go. Catdv as I mentioned before can also move your assets for you. The advantage of this is that it updates it records to reflect the new location. As a general rule its not a good idea to do stuff behind your Asset Managements Systems back. Use the tools contained within it and life will be easier, however if someone does the tools are there to fix matters

Kevin Duggan
CatDv



Return to posts index


bryson jones
Re: Catalog Limitation Organization Questions
on Nov 30, 2010 at 5:55:04 pm

Let's go you one better. CatDV Worker can move your files to a "country" folder based on filename or metadata. So if something gets dropped into a generic import folder and then you catalog and tag the asset, the Worker will see that metadata change and file the files into a folder by whatever you like.

Ex: I drop files from Ghana into your drop. CatDV imports them into a generic catalog called "11-29-2010" for instance. You then tag them as "ghana" and after that change is saved, the Worker takes all files tagged "ghana" and moved them to a folder for all the Ghana files.

If you have good naming structures, (which you should) you can have a drop that moves files by filename or enclosing folder name as well.

Ex: files in a folder named "Ghana_11_19_10" get put into the Ghana folder by default and you can even extract parts of the name and publish that into a field so that they get tagged "Ghana" in the database.

Once things are tagged, who cares what catalog they are in? You can pull queries based on any field and also filter results by any field.

That's the basis of our "Long Live The Catalog - The Catalog Is Dead" statement. (yes, that gets swapped around a bit) Once you have metadata, the catalog dissolves as far as organization and "saved searches" and filtering replaces it.

Yahoo vs. Google... Google won. No matter how obsessively you keep your bookmarks I'll bet it's almost as easy, if not more so, to just "google" something.

bryson

bryson "at" hidefcowboy.com

hidefcowboy.com


Return to posts index

David Jahns
Re: Catalog Limitation Organization Questions
on Feb 24, 2011 at 6:25:50 pm

Trying out CatDV myself, and find it odd that it has a 3000 record "limitation". I know it's not a hard limit, but if performance bogs down too much it might as well be. (I'm somewhat database savvy, and have experience with Filemaker Pro & FC Server - both of which easily handle 100,000 records or more. I understand the thumbnails are more taxing than text, etc...)

I'm hoping to use CatDV for a number of things. One of which is as a searchable Sound Effects library.

The Sound Ideas Catalog alone has 50,000 files, and we have a few other collections as well. Sound Ideas provides a searchable FileMaker Pro database, which is great, but it doesn't link to the media files. In the old days, we'd search, print out a list of candidates, grab the actual Audio CDs, load them up and listen to them. Now, we at least have the entire library in 48K AIFFs on a server. What we'd really like is to search for "gunshot" and be able to instantly listen to all of the choices. Then either copy the files, or send them directly to the edit system. Cat DV seems perfect, right?

Not if it can only have 3000 records... What's the solution? Break it into 20 separate catalogs? Very clunky, I would think - and it also means you'd have to get the Server version. BUT - audio files don't have thumbnails, right? Any idea if it could handle 100,000 records if they're only audio?

David Jahns
Joint Editorial
Portland, OR


Return to posts index

bryson jones
Re: Catalog Limitation Organization Questions
on Feb 24, 2011 at 8:56:50 pm

David, we can't explain it here but know this fact:

CatDV is based on MySQL which, depending upon who you ask, can handle up to 4 billion records. (there is some dispute online over this being smaller than the actual number possible. I suspect some of Mr. Duggan's clients will soon be able to tell us.) ;)

The "catalog" is only a base container which fades away once you have any metadata at all. A catalog can have x number of clips (3000 for now let's say). But the catalog is irrelevant.

Let me say that again. The catalog is irrelevant. No one in a large system would open a catalog to find something.

You have 50,000 sound effects. They came organized either on CD's, DVD's or a drive in folders. None of that matters, you search for "car sounds" and the database delivers a list. Or, you use saved searches or groupings to browse the server by whatever category and metadata you choose, live. Just as you found things on multiple DVD's using a database, CatDV does the same, but it also points you to the media at the same time.

There is no 3,000 item limit in CatDV Server. But just as you would leave Microsoft Excel for Filemaker when a spreadsheet got too large, you would leave the standalone client for CatDV Server when you get into large datasets. (again, a point beaten to death on this board)

If you don't have integration help, you might want to contact your dealer so that they can properly demonstrate how this works. It will make your life much easier.

Let us know how it goes!

bryson

bryson "at" hidefcowboy.com

hidefcowboy.com


Return to posts index


David Jahns
Re: Catalog Limitation Organization Questions
on Feb 24, 2011 at 10:14:26 pm

Thanks, Bryson. (I watched your CatDV demo on YouTube. nice!)

So, regarding this 3000 item "limit"...

OK - any huge catalog (like 50,000 SFX) is going to require the Enterprise Server version, right?

And its best to split large chunks into catalogs of 3000 or less items, right?

Next question - One of our other needs is for a searchable job elements archive. Our shop does roughly 100-150 jobs per year. Would it be recommended to do a separate catalog for each job?

Any limit to the number of catalogs the server can search across? Does performance breakdown with hundreds of catalogs?

David Jahns
Joint Editorial
Portland, OR


Return to posts index

bryson jones
Re: Catalog Limitation Organization Questions
on Feb 25, 2011 at 1:21:49 am

First, thanks for the kind words. The 4th installment will be up tomorrow or Monday.

Next, as far as server limits, let's be honest, unless you have more data than RedBull, Sony Music Entertainment or NASCAR (who has 3 Petabytes of proxy according to the last I heard) you're good.

Enterprise Server means that, for the Enterprise. I myself have a rig running 47,000 clips in 14,000 catalogs. Users regularly have hundreds of thousands of clips. (I remember when NASCAR had 900,000, Kevin, do you have a recent count?)

As far as catalog creation, think of them like tapes if that works. No one ever griped about not having a 30 hour video tape, right? And for 30 years we've all managed fine. The good news is now, when you need to search for a telephone ring, you can find the one's from the Europe catalog, the Asian rings catalog and the USA ring sounds catalog. Clips, not catalogs. The whole point of an asset manager is to make folders and drive location irrelevant. But it does help to have a basic structure, whether that's by shoot day, project, tape or whatever else you choose to split by. (and you can change in an instant).

But the real thing is that CatDV, as I know it, integrate it and was taught it, is a server product. Enterprise Server is the basis for all my integrations and workflows.

bryson

bryson "at" hidefcowboy.com

hidefcowboy.com


Return to posts index

Matthew Stamos
Re: Catalog Limitation Organization Questions
on Feb 25, 2011 at 7:33:08 pm

SFX files do not have multiple thumbnails so you should be able to make catalogs much larger than 3000 with only audio files. The important part and to reiterate what Bryson mentioned is the concept of a catalog though still an important building block melts away in CatDV once you have the server housing all the records.


Return to posts index


David Jahns
Re: Catalog Limitation Organization Questions
on Feb 25, 2011 at 7:43:37 pm

Hi Matthew!

Knee deep in the trial. Yes, I'm finding that even with 11,000 records in my Sound FX catalog, it's preforming quite well. Although, frustratingly, it would always crash on large imports, so I had to drag each CD/Folder of SFX in separately, which took quite a while.

I understand the idea of the catalog, but since I'm starting my demo with the standalone version, I thought I'd see what it was capable of as a single unit. So far, so good...

David Jahns
Joint Editorial
Portland, OR


Return to posts index

Matthew Stamos
Re: Catalog Limitation Organization Questions
on Feb 25, 2011 at 7:55:24 pm

David that is the recommended approach since most of the "magic" happens in the Client application. Once you get the server going this really opens up the ability to collaborate and share!


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]