A very common problem I encounter when keying is when two elements in the foreground intersect - two actors, an actor and a prop or an actor and his own body (eg an arm crossing the body). Whenever this happens I get an effect rather like pinching a drop of water between your fingers - the two surfaces appear to stick together. I've tried everything I can think of to eliminate this artifact but to no avail.
The footage was shot on a 5D Mk II so the colour info is quite compressed.
I wonder if anyone else has had this problem. I've tried searching for it on the internet but haven't found this problem described and Mark Christiansen doesn't seem to address it in his book.
When shooting, make sure that you have sharpening turned OFF, and I suggest using sRGB for the color space in the camera. sRGB shares the same primaries as Rec709, and is a smaller gamut than AdobeRGB. WHen you are shooting into a compressed and low bit depth format, such as the 8 bit video in the 5D, you want to use the smaller colorspace.
While I'm guessing (if you could upload a segment of the camera original, I could look), try adjusting the SHRINK setting under screen matte.