Re: Green Screen Dancing Edge Pixels (After Effects CS6)

Is it possible that as you work with premier (or whatever) and AE you are rendering your edit chunks out for use in AE and therefore the footage is being compressed and recompressed and recompressed? Compressing an H264 gets exponentially worse the lower the original bit rate is. And then if you recompress that, well, its starts getting very noisy. If you must render out clips to use in AE use a truly lossless format. Since it's coming from H264 even Quicktime Animation best is lossless and high quality for that kind of process (and its runLengthEncoded so green screen stuff has very low file sizes without loss).
Are you color correcting the files in Premier before they go to AE? You want as raw an original as you can get for the key - every time to color correct you loose information. Try it - take an image in pshop and look at the histogram - now color correct that image, flatten the correction (make it so!) and then look at the histogram again. Instead of a continuous set of curves of the colors you get a spiky skyline - all those gaps are missing data.

H264 can work (as I think that clip I uploaded shows - at a higher bit rate at least) but the idea is to get as good a record as you can of the on-set stuff because something will cause you grief and the cleaner the matte you can pull without any tricks, the more tricks you have left to fix other stuff.
Did the AE file I sent with the key in it not work out well enough?
You could try another keyer - Primatte makes a good one - not sure if there's a demo or the cost. There's the granddaddy of them all - Ultimatte. Sometimes another compositor (Shake, Combustion, Nuke etc) will have their own keyer that will be dialed in just right to pick out details that others miss.
There used to be a pro version of the Keylight keyer too. It could tease out that much more of your edges.

In the end though, the clip you uploaded is really noisy so there will only be so much you can do without by-hand paint repair. When it comes to fixing green screen shots, rotoscoping certain areas is not a dirty word - most of the lord of rings got roto'd even with all their blue and green screen work. And don't forget you don't need a perfectly opaque matte - its ok to have a bit of the white part of the matte (the head) be grey - you can always make a garbage matte that is rough and fills most of that back in and leaves a feathered edge being a little transparent, and being able to see hair wisps at the edges is far better than helmet head.