It may also be something as simple as caching the audio for playback too. If you've got a long(ish) work area, I've seen this take quite a while to figure this out.
Agree with Scott's comments about multiprocessing though.
All the best,
Ben
Curious Turtle Pro Video
Training | Editing | Support
Out Now - Film Wash Color Effects Vol. 3
for After Effects & Apple Color