There's a lot in your post, so I apologize if I misread or misunderstood.
Here are a couple of things to think about with regard to banding:
- Make sure that you're not checking image quality on a cheap (i.e., common) LCD monitor. Many LCDs are effectively 6 bits per channel; so, it doesn't matter if you've got smooth 16-bpc gradients in the data if the image is being quantized to much less precision on your preview device. (You didn't actually say what device you were seeing the banding with.)
- After Effects introduces dithering to reduce banding any time the data passes through an 8bpc bottleneck. E.g., if you're outputting to an 8bpc format, After Effects will dither the colors in the rendering phase. You can use this factoid to force dithering earlier by applying an adjustment layer that does nothing but uses an 8bpc effect---such as the Arithmetic effect with the default settings.
---------------------------------------------------------------------------------------------------------
Todd Kopriva, Adobe Systems Incorporated
putting the 'T' back in 'RTFM' :
After Effects Help on the Web
---------------------------------------------------------------------------------------------------------
If a page of After Effects Help answers your question, please consider rating it. If you have a tip, technique, or link to share---or if there is something that you'd like to see added or improved---please leave a comment.