Sounds like first you need to stabilize your "B-eyes" so they stay in static position. You probably need to stabilize scale, rotation and position. I'd precompose the result.
Then track the A-eyes and apply the tracking data to the precomped B-eyes.
Of course this is the general approach and depending on the details of the footage you'll run into problems that will need smoothing out.
Hope this helps,
Barend
Raamw3rk - digital storytelling and visual effects