An Island's visual database
I've been planning an audiovisual project in my island for sometime now. The goal here is to create a (large) Media Database containing video, audio and photo files, driven by CatDV software. Gran Canaria is a small island (roughly 50 Km. diam.) and the idea is to capture all of it's beauty; landscapes, villages, traditions, people... you name it.
Of course is going to take some time (I'm thinking a two years period for starters) for the database to be 'populated' We are just two people in the studio... for now.
The database is intended to serve a dual purpose:
1. Provide the Local Government Bureau, City Hall and such entities with an organized collection of ready to use, up to date, hi def media for touristic promotional material (that's our way of living... tourism) so some remote access, with controlled permissions, will be necessary. (I will call them 'political partners' as long as I pretend THEM to finance this)
2. Create e Web site open to anyone interested in getting to know our little, beautiful island. HTML Publisher will get the 'dirty job' done and a web designer will make it look neat.
I'm trying to come up with a workflow scenario first in order to design the hardware-software I'm going to need to implement such idea. Being a small, Mac based production house, we already have all the shooting gear and experience, Final Cut Studio, After Effects, Kona & Decklink, Raid disks... but I want to start from scratch on this one, I want to 'renew' the philosophy within my studio, embracing FCPX as a main editing tool (for this project at least...) which implies renewing hardware as well.
Actually, I do not foresee any shared editing to begin with... Just basic trimming and CC of masters before adding 'em to the database all from a single station. How big a database, you ask? I have no idea... I hope for a very big one as that might represent work for the next few years on my side. On top of that, I also expect those Bureaus to, eventually, ask us to put together promo pieces for international fairs and the like. In a way, I'm trying to create an opportunity for myself. So if I understand it, the database has nothing to do with the masters or the proxies file sizes, right? Only metadata (txt), thumbnails and pointers to the actual files make it to 'the database', making it A LOT smaller than the clips it keeps track of.... Is there such thing as an approximate DB-MEDIA ratio??? Our goal is to end up with anything from about 20 to 50 hours of raw video after two years.
Well, workflow… here we go: I've decided to use ProRes 422 HQ 1080 25P as the main codec for all video data. We use either a Panasonic AF101- Atomos Samurai ProRes recorder combo or the small setup: Panasonic GH2 – Atomos Ninja. The AVCHD files from the SD cards are only kept around long enough for the ProRes files to be checked. After that, the cards get formatted.
1. We go out to shoot. Comeback to the office with the shots for the day (following a pre-set schedule of important locations, specific dates for yearly events...) NOTE: for the occasional time we run-n-gun without external recorder we'll let FCPX do the transcoding before going to step 2.
2. ProRes Samurai or Ninja files go straight to a RAID in the edit machine, running FCPX (27" iMac - Thunderbolt - AJA IoXT) Then, get uploaded to CatDV in the catalog assigned to the project and let your wonderful software do it's magic at a local level first (no external access) No metadata is entered yet, no proxies are created…
3. Review the footage. The good takes are then trimmed and retouched in FCPX to become the REAL MASTERS FOR THE PROJECT and are tagged to be backed-up to LTO. The rest of the material is discarded and deleted from disk (can any/all of this be you automated?)
4. Once the back-up is ready CatDV creates proxies (ProRes proxy) of all the accepted and finished footage. Piece o' cake...
5. Next stop: tagging the clips with a very specific set of metadata tags and fields, customized for this specific project (MacBook Pro)
6. Once the clips have been fully tagged, new (smaller) proxies are generated and they are made available for everyone else. (As I understand, proxies are the files that get served to HTML Publisher to create the web clips, right? I guess you can have more than one set of proxies (different sizes, watermarked or not, different metadata sets... ) so, in my case, I assume you can have published different sets of the same catalog, one for my 'political partners' and another for general web visitors) What I'm trying to say is: can person A browse the catalog from the regular web page with a watermark on each clip, smaller size and limited metadata while person B ('partner') gets a bigger clip, no watermark, more metadata... ?
7. Apart from 'digital housecleaning' and enforcing good back-up strategies of masters, database, etc... go back to point one and start all over.
I like to concentrate on shooting nice images, not managing the DAM, so will try to iron out the steps as much as possible for it to be a mostly automated process on the logistics side.
The database will run on a Thunderbolt MacMini server into a http://www.sonnettech.com/product/xmacminiserver.html, configured to host external SATA storage for Proxy files on something like http://www.sonnettech.com/product/fusionr400p.html and backup all masters on LTO tape to something like http://www8.hp.com/es/es/products/tape-drives-enclosures/product-detail.htm.... The second slot on the Xmacmini server ‘thingy’ will host a multiport Gigabit Ethernet NIC to provide shared storage of the 'big' proxies to the other machines on a separate, isolated network. I know the Mini is limited in horsepower but like I said, no big files sharing at this point (besides, the iMac has fast, local storage just for that in case we need to edit something heavier)
Of course, CatDV will be the hub of the software side of things. I'm thinking Workgroup Server running on OSX server on the Mini, and 2 licenses of 9.0 Pro on Lion for the iMac and MBPro. FCPX for editorial and ingest, Adobe After effects for compositing, Compressor-Episode for encoding... you get the picture. Most of the editing work will be done on the iMac, then the MBPro comes in with some GFX work and metadata tagging and the Mini could pitch in sometimes for metadata entry only.
The web accessed part of things is where I'm not so clear about. . Will the MacMini server be enough to handle the web hosting also, besides managing the database and the occasional file sharing duties? Should I consider a second Mini or a used MacPro as the server?
Considering the file sizes we are talking about here (20 hours of ProRes 422 HQ 1080 25P = about 2 Tb) I'm even thinking on the possibility of having a full copy of the masters always on line for internal use only.
From here on I'm lost and that's because I know what I want but I do not know how the MAM works. I've been reading quite a bit on forums (gotta like Bryson Jones!!! jajaja) and understand all of the above is not a big deal for CatDV. I furthermore believe that the Worker node can beautifully automate most of the repetitive tasks for us.
I’m sure your experts can share some light and guidance over all this.
Maybe even tell me my whole idea is a mess and I need to start from scratch!!
Do you have an authorized dealer in Spain?
Thanks in advance
Hello and welcome to the forum.
I've asked the Squarebox crew to contact you regarding a reseller and providing a more end to end solution. Your plan looks solid it's mostly a matter of making sure it's executed well and you have the training to feed it properly since it seems to be a large and long-term project.
Feel free to contact us here again if they don't reach out soon.
bryson "at" hidefcowboy.com
Many thanks indeed for your detailed enquiry. I see from our records that I contacted you with some of our product information back in January.
There are a lot of things to talk about here, so I will telephone you to discuss this interesting opportunity in more detail.
I can assure you that you are not the only person with this workflow requirement !
Business Development Manager
Square Box Systems Ltd.
Thanks a lot Neil.
Yes, I've been dealing with the project for quite some time now... I had it almost 'sold' right before the global economy hit so it had to go into a waiting period. Now I'm rethinking the whole deal to be able to offer a new perspective, lower costs and more functionality.
As stated in my original post, this is nothing more than a project and what I'm trying to do at this point is to put together an estimate.
Hello again Neil
I've come up with a set-up of what I think the whole system should look like (but, then again, what do I know?)
Please let me know what you think about it from a hardware standpoint...
Also, I'm about to purchase a license of CatDV + CLPlus to start getting the feeling for the application with a personal project involving a lot of dv tapes recorded during a 10 year period plus some footage already digitized to HDD plus some SDXC cards... I'll keep you posted on my progress