Hey so I do an abridged series for my youtube channel. If you don't know what that is. Basically I take an anime (Naruto for example) Then I use the shows animations and insert my own custom voices. Most of the time, I need to edit characters lips to make them talk. Well for this example. This guy needs to talk, and I have to overlay a different shot of him in the track above. Then I needed to mask the frame so that only his mouth was visible. Well I got what I wanted to work. But you can CLEARLY see the outline of the mask around his mouth??? Why is that there? This has happened so many times before. I just dont understand what causes it. Any help would be nice :D
On the picture I don't see any outline ...
However, such thing could happen, if your above (masked) track is not 100% accurately placed, or has minor differences in effects parameters I guess.
A quick and dirty fix could be to blur your mask a bit (using its feather option).
If you have the time, inspect your tracks and FX for any inaccuracy.