I am testing out set extensions so I can understand how to do them correctly. On my test, I have the video motion tracked (using after effects cs4) but the object that this is applied to is very jumpy although it stays within the area that I have it placed. (it doesnt look like part of the environment as it wiggles)
I dont know if I need a better tracking point or what, I just did a short clip and will try again to see if I can get a video with better tracking points.
If it is bouncing a lot it is probably just the tracking point jumping a few pixels here and there. I have seen the same thing.
The first step on cleaning this up would be to remove a lot of the keyframes in the tracking data. By default you are tracking a keyframe every single frame. You can probably get by with a lot less, depending on how smooth the motion is. If the motion is very linear or repetetive, start taking out keyframes, leaving like 1 for every five frames or even less. This will smooth out the motion. do it slowly checking the track every time you clear out keyframes, to make sure you didn't lose some important keyframes.
This should start to smooth out that jumping look.
Unless you're wildly swinging the camera about, I doubt there's too much camera movement. It's probably just a bad track. Try retracking the shot and managing it more carefully. There is an artistry and frequently some hard work involved in accomplishing a flawless track; practice makes perfect.
What kind of camera move are you doing? There are movements that you can't do in AE and have them look good because it's only a 2d track.
Keep in mind that subtle is better. A normal hand-held camera shot instead of a tripod should be enough to sell the shot.
Also, if you're using a wide angle for the shot, you could have an issue of lens distortion at the edges of your picture; either compensate for it with optical correction effects on the original footage or the set extension. (Or shoot with less distortion by switching lenses or, if it's a fixed-lens camera, zoom in a bit to narrow the focal length.)