Motion tracking bugs?

Hi I am trying to stabilize a moving video image to key over another moving image. Both shots are hand held underwater shots. The top layer is a shot of a girl pulling a spear out of a log. The bottom layer is a shot of a whale swimming by. I want the finished result to look like she is pulling the spear out of the whale. So I Motion Stabilized the top layer and added a tracker at the precise point where the spear entered the log and hit Analyze. Motion tracked the point perfectly but when I played it back the tracker point still drifts up and down over the background image. Am I doing something wrong?