Take a look at my system below my signature. I am getting very smooth operation from this configuration. None of the anomalies you are mentioning Make sure that all the elements that support the Mercury Playback Engine are adequate. Your ram is rather low and I see no mention of a raid array. Of course graphic card support and fast processors round out the equation and not being a PC guy yet I cant speak to those.
You are running non-supported software, true, but it sounds like there are other issues going on.
Tom Daigon
PrP / After Effects Editor
http://www.hdshotsandcuts.com
Mac Pro 3,1
8 core
10.7.3
Nvidia Quadro 4000
24 gigs ram
Maxx Digital / Areca 8tb. raid
Kona 3