After Effects is a great app for motion graphics and compositing, but compression is not its strong suit.
I'll quote forum leader Dave LaRonde:
Dave's Stock Answer #3:
Don't use AE to compress files for final delivery. The various compressors are there only to make quick 'n dirty files showing a project's progress to producers, clients, the kids, etc. AE is incapable of doing multipass encoding, a crucial feature that greatly improves the image quality of H.264 and MPEG-type files in particular.
Render a high-quality file from AE, and use a different application to do the compression. Popular ones are Adobe Media Encoder, Sorenson Squeeze and Apple's Compressor, which comes bundled with Final Cut Suite. Even compressing in Quicktime Pro is better than compressing in AE.
Making good-looking compressed files is almost as much an art as it is a science. It is NOT straightforward at all. I recommend asking a few questions at the COW's Compression Techniques forum.
Principal & Designer at Keen Live
Motion Graphics, Widescreen Events, Presentation Design, and Consulting RenderBreak Blog - What I'm thinking when my workstation's thinking
Creative Cow Forum Host: Live & Stage Events