A Deeper Technical Understanding
I am deeply interested in keying in all its variances. I have read and studied just about everything I can get my hands on to better understand what's going on under the hood. While there's a lot out there, Steve Wright's book on keying and compositing is so far the best I have found without resorting to college text books on image processing. I'm wondering if anyone is aware of one or more books that basically pick up where Steve lets off and digs down even deeper into some of the engineering details. Yes, I am an engineer (retired) and math is OK. Thanks.
I have no math to offer, but I DO have information on the tricks popular cameras use to reduce the amount of data they must record. Think of the TV image is having two components: B&W & color. Color overlays B&W to create the total image.
Almost every camera cheats when it records the color component. It's called Color Resolution, or Color Sampling. There's no problem with the B&W component: for every pixel, there is a corresponding pixel in varying shades of gray from pure white to total black. However, they play fast & loose with the color component.
Perfection is 4-4-4 color: for each 4 B&W pixels the camera records, there are 4 corresponding color pixels to overlay.
But many cameras use 4-2-2 color: for every 4 B&W pixels, the camera records 2 color overlay pixels; software stretches these pixels to cover the B&W pixels. Thus, color information is compromised.
Even more cameras use 4-1-1 or 4-2-0 color: for every 4 B&W pixels, there is only 1 color pixel recorded. Color information is even more compromised.
All of these color recording schemes are good enough to fool the human eye. A computer is a different matter.
What do you need for a good chroma (aka color) key? Accurate color information. Surprisingly, 4-2-2 color works very well for keying. 4-1-1 and 4-2-0 color are different matters; the discrepancies begin to manifest themselves in poor-looking edges where subject and background meet... and chroma keying is all about the edges.
As you might suspect, a camera's price is in direct proportion to the amount of image information recorded per video frame. DSLR's and HDV cameras -- the most inexpensive video cameras available -- are at the low end of color resolution.
Color resolution is only one part of image quality. Bit depth plays a role: 8 bits of color isn't as good as 10 bits of color. The nature of compression is also significant: Some cameras' recording codecs approach a video image as a group of frames: there are two key frames containing complete image information, but the intermediate frames will eliminate redundant image information. Will such recording schemes always get it right? Well, no.
Does that help?
Although there is a confusing array of camera and codec options available, it still comes down to the phrase, "You gets what you pays for". Your pocketbook, and your personal tastes in image quality are the key factors.
I should also add this: for any chroma key, good lighting is ESSENTIAL. You can't expect to shoot with any camera with the expectation that a green or blue background will fix everything. You need to light for chroma key.
Having said that, I've enjoyed great success by shooting outdoors using Mr. Sun, but angling my portable chroma key background to be in shadow. It works well.
Former Sr. Promotion Producer
KCRG-TV (ABC) Cedar Rapids, IA
Check this tutorial about the Technology of Chromakey. You may find this helpful.
3D and Motion Graphics Artist
2011 3D Demo Reel:
Thanks both. I agree completely with your assessment. What I'm actually trying to do is better understand the actual internal math that allows keying, spill suppression, and compositing to work. Steve Wright's book Digital Compositing for Film and Video does go into some significant detail and suggests that if your keyer of choice (Keylight in my case) runs into a problem there are alternatives if the math is known. I suspect, there is still another level of detail beneath what Steve describes and I suspect I may indeed have to get into the science of image processing to wring out any more.
My advice is to look for a company in your area that does professional chroma key studio setup for tv stations, virtual studios and so on and take out the lead tech guy for a beer. I learned more technical details this way than from any book I've read and it was more fun as well.
Tudor "Ted" Jelescu
Senior VFX Artist
I learned a lot about compositing from this classic:
Todd Kopriva, Adobe Systems Incorporated
After Effects quality engineering
After Effects team blog
Todd's suggested book is a good one.
I recently had a lot of keying to do and had the opportunity to try a few different keyers.
I'm most familiar with and comfortable with Keylight, which is a color difference keyer.
Ultimate is also a color difference keyer and works on the same fundamental principal as Keylight(although with a somewhat different approach).
There's also Primatte which takes an entirely different approach to keying in terms of what's under the hood. Conceptually it uses 3 different polyhedrons that contain color values which correspond to the solid, transparent and spill colors. Most of the adjustments you make to key internally deform these polyhedrons. I was very impressed with Primatte initially but found it difficult to massage the key into something I was satisfied with.
I also tried out the Image Based Keyer (IBK) in Nuke which uses a clean plate to key. I assume this works somewhat similar to a difference key, but the results are pretty amazing. You can pull some fantastic edges even on a really messy green or blue screen. It seems that you really need to rely on another keyer to generate a core matte though.
That was a long winded post :) I'd encourage you to look for training materials for some of these keyers (some of the better ones cover the theory to some degree).
Thanks for your thoughts. I have looked at a few other keying packages but since I don't have them, I've not gotten too far. Where I am starting gain some deeper insights is in study of scholarly articles on computer graphics and research papers from several universities. Of course, it's easy to get bogged down in the math, but there is a lot going on that I suspect will seriously impact our industry one day. I'm especially impressed at some of the progress being made in keying based on depth mattes which will separate an arbitrary subject from an arbitrary background (the holy grail if you will). In its current state, this requires a laser scan along with the photography, but there are some statistical approaches showing promise. In the meanwhile, I am headed towards reverse engineering if you will the Keylight algorithms, at least close enough that I can conceive of variations and alternatives to help the process along when the going gets rough.
I hope others are interested in the details and this thread keeps going.
[Dale Paquette] "In its current state, this requires a laser scan along with the photography, but there are some statistical approaches showing promise"
While I'm sure a laser scan would be more accurate you can also achieve this with a stereo camera setup or specialized cameras that use infrared sensors. Software is of course required to generate the depth from the stereo image.
[Dale Paquette] "In the meanwhile, I am headed towards reverse engineering if you will the Keylight algorithms, at least close enough that I can conceive of variations and alternatives to help the process along when the going gets rough."
There are lots of processes involved in Keylight, which I'm sure you're aware of. The Keylight documentation should give you a good basis for what's happening in general terms. It is, at its core, a color difference keyer so the basic keying algorithm involves generating the alpha based on the relative strength of the key color against the other color components. In the case of a green screen: more green than either red or blue = transparent or semi-transparent pixel.