Are you rendering with maya software? If so, i dontt really know of any good way increase memory, far as i know, it just trys to use all that you have.
If your using MR, check under the memory and performance options, you'll see a memory limit and BSP seciton. If you use Large BSP, often times that will help out a lot. You can raise the memory limit, but that will depend on how much you have in your machine of course.
If any of your models are the same, try instancing them vs duplicating them. That helps out with memory issues. Also look in to .bot textures and .map textures. They will help you optimzie your texture memory as well. (it will only load the part of the image it needs supposedly)