I just finished
a project that I did a lot of retiming on, hit this same issue with optical flow analysis. So, before sending the retimed clips to Motion, I used Media Manager on the sequence to make a copy of the footage and trim to just the media used.
For all the flak people give the Media Mangler, it worked perfectly for this. Kept all the retiming information almost perfectly, and left only the media actually used in the timeline, so the optical flow analysis went far more quickly in Motion.
If you don't want to media manage your whole sequence, copy your retimed clips to a new sequence and just MM that. Just make sure that it's copying, not deleting your media...