controlling effects with mattes
by Dave Quirus
on
Jul 21, 2008 at 5:42:49 pm
Hi everyone,
I'm setting up a composite in AE from multiple passes done in Maya. Every thing's going smoothly except I would like to apply effects to certain layers but restrict where those effects are being applied via a matte channel. This is very easily done in a procedural compositing app (nuke, fusion, shake, etc) as you can always apply an effect or operator and it will have a matte input in which you could connect another image/chain to restrict the effect's placement.
For a simple example, let's say I have a single 3D rendered image of a sphere and a cube sitting on a ground plane. I also render out an RGB matte where the sphere is completely red, the cube green, and the ground blue. Very typical in production. Now let's say I want to use that RGB Matte pass to color grade the sphere all by itself. Obviously I want to apply a color balance or something of the sort to my 3D rendered image but I want the effect to be held out by the red channel of my RGB Matte pass.
The two different setups I have are working, but cumbersome so I'm really looking for an alternative solution to what I have that will clean my project up. What I have done is create a main comp where my 3D render sits. Then I have my RGB Matte layer in a precomp by itself with a Set Matte (or Set Channels) effect on it set to only use the red channel (the sphere). That precomp is brought in to my main comp and I've created two separate methods to have it affect the other layer. Setup #1: use the precomp at the top of my main comp as an adjustment layer and apply my levels, balance etc to alter my sphere in the 3D rendered image below. Setup #2: Create two copies of my 3D rendered image on top of each other. The bottom being the untouched base, the top being the one to be graded, balanced. The precomp is then placed above the 2nd copy and is used as a track matte for the duplicate below. Thus holding out the 2nd "effect" layer with just the red channel (now placed in the alpha) so I get a color graded sphere sitting on top of the base image with everything untouched.
Personally, I don't like either setup. Both are cumbersome and messy. The adjustment layer setup will eventually affect everything below even if it is "held back" by the sphere red channel. The track matte setup works but it's not the best practice for controlling how the edges of the effects layer will interact with the base layer. Just not very clean.
So I'm open to all ideas.. Thanks for reading such a long post. In the end, what I want is very simple and is done constantly in any procedural compositing app in production.. Apply a color balance effect to a layer and then plug a channel into that effect as a holdout matte for the effect.
Re: controlling effects with mattes by Dave Quirus on Jul 21, 2008 at 8:56:59 pm
Hey Chris,
Thanks for the response..
The only problem with 1. is that this method with adjustment layers, whether precomping and reordering or not, will become problematic when dealing with different lighting passes. For instance, let's say i have a diffuse and specular pass. I need to add the specular pass on top of the diffuse but i want to color correct just the sphere of each of those passes individually. so basically i'd have to precomp the diffuse with all the set matte precomp craziness in there. then i'd have to precomp the specular with all the set matte precomp craziness as well. then finally have a main comp to add the diffuse and specular together. so basically 3 levels of precomping for something that should be very simple.. and not very easily color corrected while viewing one final comp. i mean, it's very doable.. just a bit over the top.. :)
As far as the Remove Color Matting effect.. Is it safe to assume that this is a un-premult operation that's going on? It appears that way. If so, is there a way to premult it back after we apply an un-premult. Either way, I'm still not crazy about the idea that one would have to composite the same object over itself just so the top copy can get graded..
Am I at least correct that AE still has no way of plugging in mattes to effects? For some reason I thought I had heard support for this finally but can't seem to find it any where within the app.
Re: controlling effects with mattes by Kevin Camp on Jul 21, 2008 at 9:37:55 pm
you may have outlined this method... but to avoid compositing an object over the top of the footage, here's what i might try:
bring your multipass render with rgb matte into an ae comp (so all the layers are there). create a new comp-sized solid, name it 'red matte' or 'sphere matte', apply set matte (take matte from: 'rgb matte', use for matte: 'red channel'). create an adjustment layer and set it to use the 'sphere matte' as a track matte and add effects as needed.
but as far as using a matte layer within an effect, unless the effect has an option for adding the matte channel with in the effect, then i don't know of an easy way to do so...
Re: controlling effects with mattes by Dave Quirus on Jul 22, 2008 at 8:49:24 am
ahh, that's interesting.. kind of a hybrid of both my adjustment layer and track matte setups. it definitely resolves keeping one less precomp left in the picture which is nice.. still running in to adjustment layer issues of affecting all layers below in regards to multiple lighting passes (i.e. a graded specular pass on top of a graded diffuse pass will double the grade, etc) but that can be taken care of with separate precomps (just not looking forward to that with the number of passes i'll be running)..
i have to admit i'm surprised to see that this ability is still not available as a standard feature since it's so commonly done in other packages and has been around for some time. i realize i haven't touched AE since 6.5 but I suppose I just assumed this would have been implemented considering there's floating point exr support now..
Re: controlling effects with mattes by Dave Quirus on Jul 22, 2008 at 9:00:19 am
i'm a bit confused with the setup above and how the alpha levels is being used to glue it all together.. i understand what the effect does, just not how it's being implemented to mask out another effect. any clarification would be wonderful if you think it'll be a solution.
