Based on the description of what you want, you will need to create a separate texture for the displacement. For instance, you could:
-create a new solid with a fractal noise effect on it
-Set the noise type to "Block" and increase the contrast
-Precompose the solid (Move all attributes")
-Choose the precomp as your displacement map layer
-Where it says "Red" and "Green", set both of those to "Luminance".
If you animate the evolution of your fractal noise, you should now see it affecting the text.