Layer moves where I dont want it to after applying motion tracking

OK here's an explanation of my project.

I have in the comp a bit of video in which I'm doing a sky replacement on since the sky was so blown out. Lets call this Video Layer

I have my image of the sky in a separate layer on top. I applied motion tracking to it so it moves with the scene. All is well. Now the one issue I had is that there is a tree sticking up into the sky, so I need to get that tree on top of the sky layer or else the sky is over the tree. So I duplicate the Video Layer and put it on top of the Sky layer.

I key out the white sky in this layer so now the tree looks like its in the foreground. Ah but theres another problem. In the shot is also a goalpost that ended up being very blow out as well, so when i key out the sky, the goalpost comes along with it. So I need to replace that goalpost.

In Photoshop, I selected just the goalpost from a screengrab and imported this image to use as a matte.

I set the goalpost over the image to where I want it. I motion tracked the original video layer again and applied it to the goalpost. When doing this, the image ends up far to the upper left from where I had it. If I move the layer back into position at the beginning of the comp where I want it, figuring the motion tracking data will compensate, it ends up going all over the place other than where I wanted it.

What am I doing wrong here, and how do i keep that goalpost layer where I want it?