As an aside, is your RED One footage 2k? 4k footage usually allows you to select 1/8th resolution. I digress...
If you're having issues in software mode, It's doubtful your GPU is having any bearing on a bad memory leak.
My initial guess was, that because it was an unsupported card running in hardware mode, that Premiere interacting with the drivers in hardware mode could cause some funkiness.
Premiere is RAM hungry, well most Adobe programs are, and I have no memory issues working with RED footage which I do on a nearly daily basis with my Windows 7 system.
The next thing I would suggest would be to grab a benchmarking program like Prime95
http://files.extremeoverclocking.com/file.php?f=205 or Memtest
http://www.memtest.org/ (I assume you're on Windows from your system specs). You can stress test your RAM and processor to make sure that it's behaving normally just in general before pinning it to Premiere.
Angelo Lorenzo
Fallen Empire Digital Production Services - Los Angeles
RED transcoding, on-set DIT, and RED Epic rental services
Fallen Empire - The Blog
A blog dedicated to filmmaking, the RED workflow, and DIT tips and tricks
