How are you submitting the render command? It sounds like you ddid not set the renderer to mental ray
You're on a Mac, right? You should have the Maya terminal app, /Applications/Autodesk/maya2011/Maya\ Terminal.term and there, enter something like
render -r mr sceneName.mb
is that what you're doing?
Michiel
http://www.thnkr.com