Qualifier chatter: a possible fix
Chatter is an old problem.
Chatter in a qualified mask, as we know, happens as a result of the filter rapidly alternately including or excluding pixels in some boundary region, such as along a finely graduated luma or chroma change. From frame to frame the various boundary pixels alternate rapidly--and this makes them so obvious--or chatter.
We know there are various helpful controls to refine the qualifier settings directly and perhaps reduce the chatter area. For example, Softness, In/Out, White and Black adjustments; all can help. But the basic boundary condition dynamic and thus some visible chatter remains.
POSSIBLE METHOD to reduce chatter to acceptable levels:
Very simple: AFTER the qualifying filter, add a node to process that B&W key signal as video and make a few targeted adjustments.
This feature of processing key signals as video is available in the latest Resolve releases and is detailed at Lynda.com, DaVinci Resolve Advanced Color Grading. Patrick Inhoffer is also at MixingLight.com. Or one could read the manual. (Nah!)
The purpose here is simple: use color tools to adjust the B&W key, specifically the border regions.
A first adjustment might be to clean up the key with contrast changes, made via a steeper curve diagonal . This can be seen further to include and exclude stuff missed by the Black and White mask finesse controls on the color filter pane.
The second step might be adding one or more mid-points to create a contrast curve but preserving the changes to the in/out or black/white points of the starting diagonal. These mid-points can introduce a more gradual sloped portion along the curve. These portions can result in some broader grey borders on the B&W key.
One may find that these adjustments also aid in enlarging or shrinking the overall shaped key, which can be helpful aside from the main purpose of reducing chatter.
Then ADD large amounts of BLUR from the Blur/Sharpen tab. Note that the blur adjustment actually extends above the top of the bar scale: just keep raising the level above the bar top. The result of added blur is a key with various levels of grey border on the key. This blurred key border is our main objective.
Finally, looking at the composite result of this filtered-then-blurred key applied to the image, we see very little chatter.
Why? This reduction in chatter seems to be simply the result of blurring AFTER the qualifier has selected its (chattering) pixels, which are subsequently-- within each frame--blurred. And it seems that this does not blur the image itself in these key border areas because it is the key that is blurred, not the image pixels.
Sound reasonable??? Lemme know.
I have to say that I am not convinced this is a solution.
Unless I have misunderstood, you seem to be simply trying to cover up the noise ("chatter") by crushing the key and then applying prodigious amounts of blur.
Have you had good results with this method? Does the resulting key look completely clean?
I would have thought all you are doing is spreading the noise out over a larger area and not removing it.
It's not really possible to remove temporal noise (which is what you are calling "chatter") by means of a spatial blurring method. Temporal noise needs to be addressed with temporal noise reduction - probably the best reason to splash out on the Studio version of Resolve.
In addition, I tend to think that very blurred keys look pretty unconvincing as a general rule, not to mention that crushing the key results in removing the detail of the grey values which you would normally want to strive to keep.
But as I say, I may have misunderstood you. I'd be interested to hear more.
is this the node structure you suggested?
In that case I tend to agree with Simon,
it can be helpful at times and specific situations where what you are keying can bleed into neighboring areas without it being too notable, or maybe in extreme cases where Matte finesse tools aren't enough to clean selection..
Have you tried mastering key mixer nodes? I find them very useful in creating a less chattery key.
Considering you are good at keying, qualifier chatter is often a sort of warning flag, signaling the limits of what grading tools can achieve on a given shot. There are so many factors that can make an area less or un-keyable, some you can fight, others you must accept. Especially when working with inferior 4:2:0 8 bit video formats.
I would add that sometimes NR added to the key node will help to a degree. But Hector is right: crappy material (especially 8-bit color) is never going to yield clean keys.