I don't want to track an element into the clip but I want to use the clip itself as an element in my comp https://ibb.co/dimGSo
the movement of the background makes the blend visible and I can't key it properly to be completely visible without a rotoscope which what I am trying to avoid :)
So you want to stabilize the shot of the soldiers and composite it onto the other shot? To stabilize a shot in After Effects, you can use the Stabilize Motion button in the Tracker. Or you can track a point using the Track Motion button (or points), apply the track to a null object, and parent your layer to that null.
However, in the panning shot you showed, there is only a small part that is in both the beginning frame and end frame. Taking out the motion would involve zooming in probably over 500%, which would make the shot unusable in many ways.