RED HW/Storage and workflow advice needed
Before I get to the questions, I wanted to say that I know there are partial answers to my questions all over the CC and internet (which as be a great help), but I was not able to find the consolidated info to get more clear big picture (though I could have missed it). So I decided to post it here. I apologize if my questions will sound little stupid, but i just want to make sure that I will do a right thing.
My questions are mainly about hardware and storage requirements for RED shot feature. This will be my first use of RED so I want to be prepared. As I have understood the average RED shot 4K RAW footage for feature film will be around 2.5-3 TB with average of 3:1 shooting ratio. The backup of RAW footage is the first step, i know. My concerns are the following steps, when I start to edit and add VFX and do a color grade and mastering at the end (for cinema projection).
1. I do have to have that 2.5TB 4K footage somewhere and somehow attached to the workstation I will be editing on. So this is how I see that: On my workstation I have 2 HDDs: the first one (500GB-1TB) for the system and software and the second one for some local storage (1TB). I also should have an external storage where I hold that footage and from where I edit - like Scratch disk. My questions are : (i) what kind of external storage you advice that would be budget friendly and suitable, (ii) what is the best way to connect it to workstation, (iii) how should the drives be configured considering the size of entire footage, performance and in order to also have a mirroring so that is any disk will fail I will not have to start the work all over (i'm little concerned about RAID 5 performance wise so real life experiences would be nice).
2. As I understood the best way to get the footage to VFX software (After Effects for instance), is to use RAW (or whatever this RAW has be transcoded to if the software does not work with RED RAW), do the VFX, export to image sequence or any other lossless format and ingest it in the editor timeline before the final transfer to color grading. Is this correct? Any advices or tips? Can transcodings be somehow avoided with Premiere and After Effects collaboration? Can I use the same workflow for transfer of footage from NLE to grading application?
3. So after the grading is done (AE or Color or some plug-in) I have to render it presumably DPX (am I correct?). The thing I am concerned with is that DPX takes up a lot of space and for a feature it could be over 10TB or even more as I have been told. If this holds true I will be needing lot of hard drives and quite a budget to have hardware that could reliably hold that volume and have it backed-up and mirrored. (i) Could you guys please let me know what is the best budge friendly (to the realistic extent) way to make DPX for final package? (ii) or is it better to have the render (and maybe digital cinema package) be made by a render house? If yes how can I bring the edited, VFXed and colored RAW footage from editor timeline to the render house (I mean physically)?
I apologize for the volume of questions, but these are the issues I am very concerned with and I do not want to make a mistake. And again if I missed something important here or you have any good tip or advice, I will be more than happy to hear them.
Thank you guys in advance for your assistance and support.
You state that you do not want to ask stupid questions, and that you do not want to make a mistake, yet you are very concerned about budget friendly. The bottom line here, is that we can say "I use brand X drives, and they work great", but ALL disk drives fail, and if you do not have the budget to have a backup plan (buying more drives than you need, so you can have backups, not just RAID protection), you are setting yourself up for failure. RED productions are big budget projects, not a kid with a camcorder, making a student film. If you are a pro, you will have a budget, and have the ability to prepare for disaster (meaning that your drives fail).
The average feature film does not use 2 - 3 TB of storage - every project is different. You can't base this on someone elses projections. The real rule is, whatever you budget for, you will need more storage - you will run out of room. It happens every time.
You write -
My questions are : (i) what kind of external storage you advice that would be budget friendly and suitable, (ii) what is the best way to connect it to workstation, (iii) how should the drives be configured considering the size of entire footage, performance and in order to also have a mirroring so that is any disk will fail I will not have to start the work all over (i'm little concerned about RAID 5 performance wise so real life experiences would be nice).
REPLY - RAID 5 is the way that all professionals work (or RAID 6). These professional products, advertised right in front of you on these forums on Creative Cow, start at around $5000. You are not going to find what you want at Fry's or Newegg. You are not going to edit a feature film using a couple of $90 1 terabyte internal hard drives, and have no issues. You need a big drive array. In addition, you need to have your original RED media backed up on even MORE hard drives, in case something bad happens to your original footage. you will not edit with the raw .r3d media, so these drives, and their backups will sit somewhere. when you run RedRushes (for example) so you can transcode your media so you can edit, this will live on your editing RAID array, which is almost 100% of the time, an external drive array on your MAC Pro.
YOU WRITE -
Can transcodings be somehow avoided with Premiere and After Effects collaboration? Can I use the same workflow for transfer of footage from NLE to grading application?
REPLY - this would be news to me, but I am no expert. You have to transcode with some program, like Red Rushes. Not unless you have access to an expensive system like Assimilate Scratch.
3. So after the grading is done (AE or Color or some plug-in) I have to render it presumably DPX (am I correct?). The thing I am concerned with is that DPX takes up a lot of space and for a feature it could be over 10TB or even more as I have been told.
REPLY - you are damn correct about this - DPX files take up TONS of space.
If this holds true I will be needing lot of hard drives and quite a budget to have hardware that could reliably hold that volume and have it backed-up and mirrored. (i) Could you guys please let me know what is the best budge friendly (to the realistic extent) way to make DPX for final package? (ii) or is it better to have the render (and maybe digital cinema package) be made by a render house?
REPLY - there is no such thing as a "budget friendly" DPX package. This is all hi end stuff, that costs a lot of money, because you need so much drive space. Simply don't own the equipment, and rent what you need, or go out of house to do it.
If yes how can I bring the edited, VFXed and colored RAW footage from editor timeline to the render house (I mean physically)?
REPLY - it is typical for editors to transcode the R3D media to another format (typically ProRes422, etc.), edit your movie, and then go to a conform house that owns the right hardware (like Assimilate Scratch) to color grade and output your movie to DPX.
I fully understand how expensive all this stuff is - you don't have to own everything - very few people do. Make sure you backup your original media, because you will need those RAW R3D files to do what you want, and you don't want to accidentally drop those drives on the floow without backup.
As Bob says "you don't need to own" all this stuff
In the long run the offline /online premise works well here. If there is someone in your area who can help you you will be better served and your budget will not suffer unless of course it is impractical from the beginning.
We use an assimilate scratch system. we generate Burned in offline footage the client can cut on a laptop if necessary, they send us the project and a quicktime of their edit, we prep the EDL for scratch, verify the conform from the quicktime ref, and deal with whatever the situation demands for color correction, framing etc. If VFX are required we export DPX or tiff to whatever the VFX house is looking for.
There are several people worldwide who can do this. This is what Scratch was designed to do.
to play back DPX in real time requires a hoss of a disk system. To manage a project requires a lot of knowledge and experience. This is where people who own these systems and do this kind of work can be a great advantage to your work.
Assimilate is a great company. They would certainly be able to tell you who in your neck of the woods has a system enabling you to contact them and discuss your situation.
Editor / Engineer
Rock Creative Images
[David Chabashvili] "with average of 3:1 shooting ratio"
That's a very low shooting ratio. A typical "Hollywood" feature will have a shooting ratio ranging somewhere from a low of 5:1 up to 20:1.
Post production is not an afterthought!
One of the projects we work on a monthly basis is anywhere between 13 to 15 minutes long. These are very precisely thought out shoots where there's no waste of money nor time and the shooting ratio is 6:1 which is very low. A shooting ratio of 3:1 for a feature is unachievable. No way. Pretty much that is 3 perfect takes for every shot. Impossible!
Then add everything else you mention, FXs, DPX, Storage, Transcodes, Finishing, etc, etc. Wow! Not that you can't but if you are trying to budget for the very minimum expect a disaster of an outcome.
Colorist / Post Supervisor
West Los Angeles, CA
- Visit our studio's website.
- RED Rocket Conversions.
- Follow us on Facebook. Become a fan!
- Duplication, Mastering and Teranex Conversions.
What do you mean by disaster - its going to sum up to huge sums or in terms of lots of technology to manage and look after?
to me, a disaster is that you shoot on your RED FW800 drive, and the drive dies. That's a disaster. If you are shooting a student film, it's no big deal, but if you are shooting ANY professional job, where people are PAYING YOU MONEY, or you have paid actors, built sets, and need to actually deliver a product, if the drive fails, this is A DISASTER. So you spend whatever it takes to make sure that disaster doesn't happen. I don't know what you are doing, and I apologize in advance for making a stupid analogy like this, but if you say "hey, we are not made of money - I just don't have $400 lying around to buy another drive as a backup" - then you should not be on this forum (or reduser.net). It costs A LOT Of money to do a professional shoot - be it a feature, a commercial, a concert film, or even a corporate video. Getting backup drives is not a good idea - its ESENTIAL, and for a feature, required by insurance companies (who may also insist on an LTO4A tape backup of your media).
It all of this expensive - YOU BET IT IS. Is all of this very technical, and complex, and a pain in the butt - YOU BET IT IS - that is why professional forums like this exist, and this is why we do all this stuff, so we can charge money for all this complex stuff. Dealing with all the bugs and "beta" software while you are trying to do a big job is a headache. Just look at the post directly above this one (the next thread up, which is current as of 2/18/2010) - the poor guy that is having a nighmare importing RED files over 2Gigs. I KNOW THIS PROBLEM, I have seen it. This is his disaster right now. And you know what - there are LOTS of disasters when you work with technical complex equipment, and new workflows like RED. You must be prepared for headaches, and having redundant storage, in case of drive failure is critical to a modern production with RED. (and anything else that competes with RED, like Arri D21, Genesis, F23, etc.).
Guys, Thank you very much for such thorough explanations of storage matters and related risks. I hoped that RED workflow would be more forgiving, but I guess its not the deal :) So I will be outsourcing the technology stuff for the film and will concentrate on film on my behalf. Once again thank you.
Working with RED footage is not complicated if you shoot short form or commercials. Even docs can be managed without spending a fortune on drives. What's makes it complicated in this case is that you're doing a feature film and outputting to DPX.
Don't want to scare off any people new to RED who may think every RED post job is an enormously expensive pain in the butt, it isn't.
[Sam Roberts] "What's makes it complicated in this case is that you're doing a feature film and outputting to DPX. "
I tend to think that the only think more complicated is the fact that it's a feature film and things always take longer, but the basic principles still apply. Having finished features shot on RED (with Scratch:)), I've got to say if you have the right tool for the job, it's a very, very easy workflow.
Colorist/Digital Cinema Specialist
Salt Lake City, UT