You're going to need more RAM too. If you're going on the cheap, get two 4GB sticks. Or whatever you can afford.
After that, I think you can only use the Radeon 5770. It's debatable whether the 5870 will run in the 1,1. I have the 1,1 and the 5770 makes it useable for X. I have 16gigs on that machine. Even with that setup, you'll be best off running in better performance playback mode, which softens everything. Mostly noticeable on graphics. You'll want to run off the internal sata drives, or off an esata external for best performance too. FW800 is ok, but that machine needs all the help it can get.
I bought my 5770 off eBay a few years ago for a couple hundred bucks. Don't know what they cost now. Hopefully a lot less. Mine was a PC flashed to Mac one. Works fine and has more outputs than the Mac version. Mine has an HDMI out that is convenient.
Thanks so much for the info. I don't need the machine for anything other than offfline editing and exporting the mp4s from our Drift innovations cameras. Usually long timelapses and on-vehicle mounts.. Do I really need more RAM for just that?