Think of a gradient or feathered vignette like a ramp -- a smooth surface rising from low to high over a certain run. However, computers, being digital instead of analog, don't do smooth. Think of a computer's representation of that gradient as a staircase, with a series of discrete steps from low to high over that same distance.
If you use only a few steps, they must be very deep to get you all the way across the run, and they must be very steep you get you all the way up the rise. As a result, you get big bumps going from one deep and steep step to another.
If you use many, many steps, they must each be shallower to fit within the run and shorter to fit within the height. A great many very shallow, short steps approximates a smooth ramp.
"Bit depth" or "color depth" is the number of steps. Because of the way computers count, using 8 bits gets you only 256 separate shades of gray between black and white, but using 16 bits gets you 32,768 separate shades of gray.
Practically speaking, just bumping your project's color depth from 8bpc to 16bpc should improve the banding, as AE will then attempt to dither the gradient (adding noise to smooth it out) when outputting to an 8-bit format.
ProRes 422 (HQ) is a 10-bit format, so it can accommodate much smoother gradients than 8-bit video can (1024 shades of gray from black to white versus 256). Make sure to set your output module's depth to "Trillions of Colors" to preserve this color depth.
Walter Soyka
Principal & Designer at
Keen Live
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
RenderBreak Blog - What I'm thinking when my workstation's thinking
Creative Cow Forum Host: Live & Stage Events