controlling effects with mattes

Hi everyone,

I'm setting up a composite in AE from multiple passes done in Maya. Every thing's going smoothly except I would like to apply effects to certain layers but restrict where those effects are being applied via a matte channel. This is very easily done in a procedural compositing app (nuke, fusion, shake, etc) as you can always apply an effect or operator and it will have a matte input in which you could connect another image/chain to restrict the effect's placement.

For a simple example, let's say I have a single 3D rendered image of a sphere and a cube sitting on a ground plane. I also render out an RGB matte where the sphere is completely red, the cube green, and the ground blue. Very typical in production. Now let's say I want to use that RGB Matte pass to color grade the sphere all by itself. Obviously I want to apply a color balance or something of the sort to my 3D rendered image but I want the effect to be held out by the red channel of my RGB Matte pass.

The two different setups I have are working, but cumbersome so I'm really looking for an alternative solution to what I have that will clean my project up. What I have done is create a main comp where my 3D render sits. Then I have my RGB Matte layer in a precomp by itself with a Set Matte (or Set Channels) effect on it set to only use the red channel (the sphere). That precomp is brought in to my main comp and I've created two separate methods to have it affect the other layer. Setup #1: use the precomp at the top of my main comp as an adjustment layer and apply my levels, balance etc to alter my sphere in the 3D rendered image below. Setup #2: Create two copies of my 3D rendered image on top of each other. The bottom being the untouched base, the top being the one to be graded, balanced. The precomp is then placed above the 2nd copy and is used as a track matte for the duplicate below. Thus holding out the 2nd "effect" layer with just the red channel (now placed in the alpha) so I get a color graded sphere sitting on top of the base image with everything untouched.

Personally, I don't like either setup. Both are cumbersome and messy. The adjustment layer setup will eventually affect everything below even if it is "held back" by the sphere red channel. The track matte setup works but it's not the best practice for controlling how the edges of the effects layer will interact with the base layer. Just not very clean.

So I'm open to all ideas.. Thanks for reading such a long post. In the end, what I want is very simple and is done constantly in any procedural compositing app in production.. Apply a color balance effect to a layer and then plug a channel into that effect as a holdout matte for the effect.

Thanks everyone,
David