I haven't seen this before but I don't use displacement mapping too much. I'm getting some pretty dirty edges when I do a displacement map on an vector image (from illustrator).
It's a pretty basic text element on top of a silk backdrop. Displacement map applied to make the movement of the silk apply to the text on top. Problem is....the displacement map dirties up the edges of the text pretty bad.
The text begins to essentially break down around 15 pixels of displacement. Any thoughts on this?
you could try working in a higher bit depth. the greater the bit depth, then greater the levels of colors/lightness that you'll have to generate the distortion, so that should produce smoother results.
if you silk footage is at a lower bit (say it's 8-bpc, and you set ae to work in 16-bpc) then you'll need to add a little blur to the footage (precomp the footage and use a blur effect, then use the precomp as the displacement map). this will produce greater levels of color, otherwise you will still get the lower bit depth issues.