I am trying to monitor my core audio through a 4 channel mackie mixer. I have a stereo mini to rca cable that i attached to the headphone port on the front of the mac pro and it is going to channels 3/4 rca "tape input" on the mixer. The audio comes across very garbled or with static. The cable is brand new. My system preferences are set to built-in out for audio. Am I going about this in the wrong way?
The setup you're describing is not optimal. The output you should be using on the MacPro is the Line Out, which is at the rear. It's the one with a little speaker symbol next to it. That output should give you more level, which should help.
The other consideration is the cable's length. The type of cable you're using is correct, but such a cable is unbalanced. Long story short, if your cable is too long (over 15 ft) and/or is in an environment with a lot of electromagnetic interference, the cable will pick up interference and transmit it down the line with your sound. With the connection you're trying to make, the only real solution is keeping the cable length as short as your needs permit. Try changing outputs before futzing around with the cable, though, as that could be the ticket. You may also want to try plugging another source that you know works into that input on the mixer to eliminate that as a cause of your problems.
If it doesn't work, report back and we'll go to level 2.