First off, that sounds like a really big render, i'm wondering if you've got something in your scene that causing a memory leak or something. I'd make sure your scene is optimized.
By defautl, 32bit windows can only allocate 2gb of ram per app. There is a switch that lets you allocate up to 3gb, but they reccomend you have 4gb of memory for this
http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx
The only other way to get more memory for an app is to run a 64bit OS, but even then you need a 64bit app, which maya is not (yet).
chad
