Setting memory limit in maya file (mentalray object, batch render option dialog, cmd (-mem)) doesn't change a thing, still mental ray keeps getting more memory until it runs out of it and crashes.. why doesn't the limit memory work?
There could be a couple different things going on.
First, you could simply not have enough RAM to render your scene. Some aspects of MR aren't disk-swappable, basically. How much are you working with? Can you post a screenshot of your scene or description as to its complexity?
Second, are you using instacing or proxies? If so, you'll want to be using BSP2 and set your Primary Render to "Raytracing" (default is Scanline).
And lastly... Are you using Maya 64-bit in a 64-bit OS? This will be key to using more than 3GB maximum of your RAM.