Unwanted: Snapping footage to tracking point

Hi there,

I'm playing with motion tracking to patch over part of a hand held camera shot, I've manipulated the photo to create a patch which seamlessly masks out the part of the shot in question.

Now when I try I use motion tracking on the footage and set the target to the image layer above the footage (the patch), After Effects snaps the location of this patch to the tracking point I used.

EXTREMELY frustrating! I've already carefully located the patch over the correct area of the footage on the first frame, but AE insists on snapping it to the location of the tracking point.

How can I get after effects to apply the tracking corrections relative to the patches location on the first frame?

Thank you so much, I really appreciate any help you can give me. I'm currently at hair pulling stage!