Have you tried applying one of AME's presets without altering it? If that gives you stuttery output, it could maybe be a problem with the speed of your system - drives or processor. H264 is a processor intensive codec (as far as I know) ie. it takes a lot of computing power to decompress each frame, which is why codecs as good as H264 are a relatively new invention.
If one of the preset setups give you OK output, start changing the settings one parameter at a time, then do a test compress, see if it's OK, change the next parameter...until you get stuttery results. Then you can isolate the problem to one particular setting.
You might get more help on one of the Adobe forums (such as
http://forums.adobe.com/community/ame ) - this is going off-topic for an After Effects forum.
- Paul