The filter is displacing the images that you have -- it can't create additional images of the guy walking horizontally for displacement, if that makes sense.
If you had shot this at, say, 1000 frames per second instead of 30 frames per second (a much more expensive project), then you'd have enough images to get a very smooth transition.
If you want, you can fake more in-between images with Timewarp: Slow down the footage and create interrim frames using pixelmotion. I'd suggest you render that out and reimport it into AE and try the time displacement again. (You don't want to do it all in AE without the pre-render, as Timewarp would have to do its work on a lot of frames, and I imagine it would take a heckuva long time.)