If the GeForce 8600GT is the card you're still using, it doesn't appear anywhere on the Sony site as being a supported one so my guess is that's your main problem.
http://www.sonycreativesoftware.com/vegaspro/gpuacceleration
Try disabling GPU and see if that helps at all.
Note: even though I have it, I'm not a Pro 11 user yet due to all the reported issues.