I'm experiencing something that's never happened to me before. I hope someone out there can give me some clues as to what's going on.
I'm using a Canon Legria HFS100 for my amateur video projects. I shoot in HD, but I have to transfer the footage in VoltaicHD to be able to edit on my PC. The image quaity is still fine for my projects, and much better than standard DV. I shoot quite a lot of my footage in fron of a greenscreen, lit with photo floodlights using diffusion screens. This usually gives very good keys. BUT whenever I shoot with people wearing black or grey things turns sour.
I pull my keys in AE CS3, using Keylight. Right now I'm working with footage showing a guy dressed up as Gandalf -all grey, that is. His face, belt and other parts NOT grey look fine. His grey clothes are an ocean of flickering grey tiles. I'm not talking small pixels here. Huge tiles are floating around on his tunic!
I've shot footage for an earlier project with people wearing black, and I experienced some pixillation problems there but nothing like this! Every thing else I've shot in this project with the Canon Legria (in SP mode) and imported into AE looks great, but the grey clothes in this greenscreen footage is seriously messed up!
AVCHD is not really great for shooting chroma. Since you convert that footage even further you loose more color info. I would recommend using the original footage to key in AE and then composite- render out the final and replace in your cut.
Also, with grey or black you may need to pull multiple keys to get a clean looking subject. One thing that has helped me with the look of the footage shot on DV/HDCV or AVCHD is using a filter like Remove Noise before you key. Also, Red Giant Magic Bullet- Denoiser can help.