I went back and tried a couple of things. Toggling settings does something but what I've really noticed is that the audio performance is terrible when there is only audio in the timeline. It's also bad playing an audio track in the preview monitor. The audio would play one out of three times and would sometimes just go mute after a couple of seconds even though the meters are still moving. I also found if I took just random video and put it with my audio only track the performance when to what you would expect. Kind of an issue since often you cut audio first and add video later.
Robert Brown
Editor/VFX/Colorist - FCP, Smoke, Quantel Pablo, After Effects, 3DS MAX, Premiere Pro
http://vimeo.com/user3987510/videos