Chroma key - problem with shimmering in hair
THE PROBLEM: I used to do a lot of green screen shooting & editing & it all seemed to work easily, but recently I did a job & I had tremendous problems. The most troublesome issue was the shimmering effect I kept getting at the top of the hair & around the upper edges.
I just spent several days trying isolate the problem with some results.
SAMPLE: Here's a rough sample of what I'm talking about. Admittedly, this has had no work done on it (one press Primatte), but even after I spent hours trying to eliminate the shimmer...I couldn't reduce it completely via After Effects.
Editing: After Effects with Primatte plugin
Lighting - greenscreen - fluros (both horizontal & vertical around a 5m screen) - daylight
- foreground - 1200 LEDs (had to use a red head & softbox for added light) - warm
Cameras - Sony Nex VG20 DSLRs using AVCHD & direct capture 4:2:2 & motion jpeg
1. I suspected that it may be due to the amount of gain needed for the cameras. The Nex VG20 has a big sensor which is supposed to compensate for the slow lens & poor low light. I actually struggled to get enough light even with 14 fluros & 4 foreground lights. Most of the time I had to use gain as high as 9dBs
Possible solution A- I tried using zero gain and more light. I even used a fast prime lens. Although it did have some effect, it wasn't significant enough.
Possible solution B- I added a denoise effect before the Primatte keying. This also helped a bit, but increased the rendering time enormously.
2. Positioning or intensity of back light.
No matter what I did with the backlight, it made little difference....even no back light at all.
3. Too much or too little foreground light.
Again this made no difference after much testing.
4. Too close to the green screen or caused by reflective spill.
The mannequin was 4 metres from the green screen & it still showed no improvement. We even covered the floor & all excess green with blacks.
6. Video compression. So the next possibility was the camera & compression. It's strange, because I'm sure that I've achieved a perfect key with these cameras before, but on other threads it was suggested that compression will cause issues especially with DSLR cameras.
Solution - . Capture footage directly & use 4:2:2 format. This did help considerably, in fact I did manage to overcome most of the shimmering, but only when I used it in conjunction with a very fast lense.
If I captured footage with a slow lens then it still had the shimmer issue.
So the solution is to use low compression & a fast lens, however, I still believe that I managed to produce good keying in the past without either of these & I don't understand why.
Any suggestions would be most welcome!
Proper lighting from head to toe is the first stage, which you seem to have sorted. Same for using a fast lens to make the chroma background more uniform.
On the post proudction side of things, don't expect just one pass of Primatte or Keylight to work. First make a garbage mask based on exactly what you need, which gives the plug-ins less to worry about. Punching up the green values before keying often helps too and then, you often need to luma matte the footage to a black & white version of itself, and then use that for colour correction and alpha matte-ing in a second or third comp.
Here's a good starting point project which should be of some help to you....
I'm not sure how a "fast" lens will help with uneven lighting. Uneven is uneven and opening up your fstop (in other words: a faster lens) will make the dark spots brighter but will also make the bright spots even brighter and the lighting will still be uneven.
Color correcting the screen color (punching it up) before keying is also a no-no. Any time you color correct something you are throwing information away. Look at the the histogram of any image. Now color correct that image. Look at the histogram again. You will notice gaps where information, (color information) is now missing and the smooth transitions from one color to another are now steppy.
Back in the optical days we had to hit a specific color of screen to make the chemical process of the film work. But these days you can key off any color. The saturation of the color and the differences between the screen color and everything else are what's important, not the actual color itself.
That shimmer you are seeing is digital noise. This is probably caused by the compression of the footage either in the camera or when it was compressed after the camera (to make it a manageable file size to send). It could also be noise from the camera's chip. I will concede a faster lens can help the noise problem out a little: when a CCD doesn't have enough light work with it injects voltage in to the chip and you get more noise - a gopro camera does a beautiful job in full sun but looks noisy noisy noisy at night. So a faster lens will let more light in. But in a green screen scenario you should have more than enough light to work with.
Always key with as uncompressed footage as you can. HD (in 8bit) is 186 mega bytes per second. If you are using H264 you are, at best, probably running around 20 mega bits per second. That's a factor of about 70! That's a huge amount of information that has been lost. And most of that lost info is going to be the subtle stuff that you need to separate the hair from the green screen.
But not all cameras shoot uncompressed (or have interchangeable lenses) sooo.
Key what you can and then compress the key a little further and use a dechatter effect to help out (if needed). The goal on this first layer is to create a smooth edges key (as in: doesn't chatter over time). This won't let the fine wispy hairs through but it will keep the edge from shimmering. It will look like a helmet head key from a B grade scifi movie.
Now on another layer do a rough garbage matte that just lets you see the edge of the hair but not the core of the person and overlaps the compressed key you did in the last step. Don't use a key puller effect on this garbage matte layer but do use a key color suppressor, so you end up with colored hair tips on gray. Color correct this layer to lighten the gray and then overlay this layer on the rest of the comp with a "darker color" blend function. (or you can crush this layer even further to make a matte for the hair and pass an uncolorcorrected hair layer through this matte.)
You can also remove the color from this hair tip layer when you start shifting levels around. Using extreme curves or levels on an image with color creates all sorts of noise, where doing the same on a grayscale image does not. With subtle little things like hair tips, surprisingly, they don't always need to be in color, or you can tint them after the fact. (when we used to do photo retouching by hand, small imperfections were always done with gray scale paints, never color. The eye better at discerning brightness than color so things like hair tips aren't noticed if they are the right brightness but not in color.)
Thanks for all your advice & suggestions.
I think that the fast lens only helped in providing more light without the use of gain, although my testing results were not always consistent.
I will certainly try the dechatter effect and the techniques you suggested. I actually have 2 shoots that I was thinking of reshooting because of the shimmering problem. Maybe I'll give your ideas a try...and maybe avoid having to reshoot (particularly as one of my actors is not available).
Thanks for the template Ciaran!
I probably do need to go back & do a few more tutorials on the keying process again as it's been a while and it's quite a complex process.
I must admit that I did try a few video tutorials, but they were extremely basic and it was a matter of them throwing in perfect footage and getting a fairly instant & perfect result.
Thanks for your help and suggestions.
Well I have spent days trying every conceivable technique possible to eliminate the shimmering hair in my footage (not referring to the samples here) and I have been unable to reduce it significantly at all!
I don't know what has caused it, but I think it is just too bad to chroma key.
If anyone wants to prove me wrong, here is some sample footage (50MBs)
I didn't finesse it but two Keylights for two different ranges and the advanced spill suppressor would be a pretty good start.
Also a lightwrap might do some good when you have the background ready.
Thanks Tero, I'll give your suggestions a whirl. I tried just keying the top of the hair all on it's own & I could not even get that happening. It just won't stop moving even when I shrink the matte right back.
It's also good to remember that sometimes half-ass is the proper amount of ass. If you have badly lit, compressed footage then you should make it look good overall instead of trying to fix some small part that viewers probably won't care about.
Will the background be white as in your original sample? If not can you supply a frame of the BG? What it's like determines much of what the key must accomplish.
Been looking at the footage.
I think you are suffering from the quality of the camera and the low bandwidth of the codec.
Even though the footage appears to be 50fps progressive, there seems to be an interlace "cover up" going on there. Some cameras only shoot interlaced video or a goofy pulldown to give it more of a cinema look, and then do a wonky pulldown maneuver to simulate progressive. If you watch the highlights in the eyeballs they are clean for a few frames and then there is this duplication that happens for a few frames (the highlight suddenly has two iterations of itself in two slightly different positions) and then the pattern repeats. So even before you get your file out of the camera, the camera is already mangling the image. To be fair, if this is whats happening, its a pretty good conversion, but its going to be part of the jitter problem along the hair. It would have been better though to shoot with pulldown, take the pulldown out manually, do the key and then put the pulldown back in if needed.
The footage is pretty noisy too, there's the sensor noise but there is also the codec noise which makes details kind of jump around (the noisy matte along the hair line).
Still playing with it here.
If you don't need the final at 50fps (which I doubt you do) bringing it down to 25fps will half the "boiling" of the hair (yes, that is the technical term for the problem, really!)
Here's a go at it with native effects.
First, in this image you can see the problem with lighting. There is a halo where the keyer has been tuned to the color right around the head but then another key must be done for the area near the frame edges where the green is darker. I have not bothered to do that in this comp and will leave that for you.
We've tried all sorts of tricks but it comes down to simply crushing the matte to get rid of the of the jittering edges. Noise is noise (especially compression noise). As helmet heads go this one's not too bad, but its not quite there in my books.
Here's the comp (AE2017)and a final output of 5 seconds worth.
The other issues with crushing the key are that you get what we call keyholes: areas where two objects come close together and the matte tends to close up before they actually touch (his hand on the photocopier for instance) or you get rounded corners instead of sharp ones (his armpit). This is solved by doing one key for the head that gets crushed as I've done here, and then another for areas that are not as noisy but with less key crushing, and then another key for the outer darker screen areas.
I used a Sony VG20 which is a semi-pro DSLR video camera. I agree that there is a fair amount of noise. I found that when I was shooting, I had to flood the subject with considerable light because the camera is very poor in low light and the lens that is provided is a slow f4 zoom lens. I even had to resort to using gain because the 4 lights I used on the foreground were not enough (3 X 1200 LEDs + a red head with softbox). The green screen was independently lit with fluros.
Sony claims that because the sensor is so big, that it makes up for the slow lens...but I beg to differ. I can resort to using my fast prime lenses but they have no zoom.
I was going to go for white after I saw how clean it looked, but a colored background may be smarter in this instance to maybe hide some of the weakness. I did another video previous to this, where the hair caused the same problem and I used a background to distract, but it didn't really achieve much.
I'll find a background for you anyway.
Thank you Steve for all your work. You managed to get the hair tamed down a lot more than I did. The rest of the issues are easily fixed... as you said by separate keys.
I think it comes down to the quality of the footage. I never thought about the 50fps....good thinking. I usually shoot in 25fps which may have helped avoid this previously.
Now that I know that it isn't so much a keying issue I'll focus on the capture again & try to improve that. I can resort to direct capture in 4:2:2, but it makes the process a bit more complicated & the files much bigger to handle.
One other possibility...my green screen lights are blue & my foreground lights red...could this cause weird effects do you think?
Again...thanks so much Steve for your help. I have spent days on this so your input & assistance is greatly appreciated.
Perhaps sony means that the bigger chip lets them get more tech in there, but on the surface: a bigger area to fill means you need more light to fill it (unless I missed the current administration cancelling the inverse square law).
If you shine a projector at the wall, the bigger you make the image, the dimmer that image gets.
But the problem remains: using that DSLR for regular shooting may be fine, but when it comes to FX work, often you need a better solution. Even in the film world we use a special stock (different from what they used in principle photog) when doing fx work to reduce grain so we can pull out the details we need. And when budget allows, even a special camera that lets us have a larger film frame, and therefore a smaller grain structure per square inch of image area when transferred to digital.
Then there's the file fomat the camera spits out. I don't think the camera noise is killing you (and we have noise killers that can overcome that), its the compression artifacts. When you get right down to it, compression artifacts are little areas of the image that are approximations of what should be there. If you can get that compression "block" small enough you end up with pixels: as close as you can get to an accurate rendering of the original. But as the blocks get bigger (and to keep bandwidth low the blocks are always bigger) they just average what they think is important. And that averaging happens every frame (or worse every, keyframe). The idea is: don't change the pixels between keyframes if nothing in that area is moving. This is a great way to save drive space but a lousy way to have a clean image. So you end up with details jumping around as the system re averages what its seeing.
Often these cameras will have a digital out: you can plug in your computer or drive and capture or record a higher bandwidth master for VFX work. Even the big sony cameras compress the image to a tape. But the feed off the back is a glorious, nearly uncompressed, 10-16 bit version, far superior to what's on the tape from the very same camera.
Rule of thumb: don't digitize after the fact; capture live as uncompressed as you can.
My last post got delayed somehow but was composed before your last.
Re red and blue lights. Do you mean actually red and actually blue? or just their color temp (3500 k vs 6500k perhaps?)
The green screen looks good to me (save the uneven lighting, but often that is unavoidable, and easily dealt with as we have discussed, especially with no action at the edges and good garbage matte will fix the lighting flags etc).
Having more red on the talent is a good thing because that will increase the separation required from the green and also helps to offset the green spill from the screen itself. If the lights were used the other way around it might not turn out as well.
Double check for noise in the reflective surfaces like the photocopier on the left. Normally we would use the green sheen to push in color from the background but with the noise problem I would just crush the matte in these areas and let it be.
Large files are the cross we VFX types have to bear. The larger the file, the easier the key, so the time spent transferring files is saved in the composting.
The main issue is that compression. All other considerations seem to be well within specs for pulling a decent key.
More light though will reduce the voltage injected into the camera chip -which is what causes the noise - more light=less noise. Less noise coming from the camera also reduces the compression artifacts - the more noise the codec sees the more averaging it has to do.
But even though you might be able to get more lights, don't overexpose - shoot for that color of green in camera - it's pretty much spot on.
There are also some fallacies out there that the talent should be over or under exposed. Neither is true. The talent should be perfectly "exposed" as should the background screen. This old optical technician's wife's tale is probably from that optical era, when every step of the process added noise and contrast. (ah I remember it well - I love the smell of developer in the morning!)