It sounds like you're simply bumping into the same limits EVERYONE bumps into: RAM Previews in AE use all the memory available, and your RAM filled up before the end of the comp.
You can use an entire litany of ways to overcome that limitation, all of which are described in AE's online help: trim the work area, preview every other frame, preview at half-resolution, preview only part of your frame.
You may not like doing that, but hey -- everyone's had to do it. It's simply part of working with AE.
Sr. Promotion Producer
KCRG-TV (ABC) Cedar Rapids, IA