ADOBE AFTER EFFECTS: Forum Expressions Tutorials Creative Cloud

RAM, CPUs and render times

COW Forums : Adobe After Effects

<< PREVIOUS   •   FAQ   •   VIEW ALL   •   PRINT   •   NEXT >>
Larry Adlon
RAM, CPUs and render times
on Sep 13, 2013 at 2:43:47 pm

Hi. I've been reading up on this subject for a while now, but am still a bit confused as to how to best set up my particular system. Currently, I'm getting radically random results from the same scene (sometimes it's 2 hours, sometimes 30!)

First of all, my specs:

Win7 Pro / i7 / 32gigs RAM / Solid state C drive, SATA secondary drives (projects and source)
AfterEffects CS6
Everything all up to date

Okay, so I obviously have plenty of RAM, yet I'm also aware that supposedly you can have settings that actually slow things down by doing things like overscheduling the CPUs, or bottlenecking the writes, etc.

I'm trying to figure out what my sweet spot is.

I don't run any other programs while using AE, so I don't require RAM/CPUs reserved for other apps set in AE (however, I'm a bit concerned that maybe I should?).

I usually have 3gigs RAM reserved for externals (as that's as low as I am allowed to go), and 3gigs per CPU, giving me 8 CPUs actually being used.

For the most part, that works great, but today, my project suddenly has decided to go from 2 hours rendering to over 30. Something obviously tripped up. Often, I find that purging the memory before rendering helps a lot, but today that didn't help.

Meanwhile, at the bottom of AE, it's saying that 5% of 32gigs of RAM is being used, which is kind of depressing. Is AE limited to a max of 3gigs per CPU, or can this be boosted? Is 3gigs overkill for a standard def project (although it has many layers and a number of pre-comps)?

I've been trying every combo I can think of (2gigs per CPU vs 3gigs... disk cache on/off, giving various amounts of RAM and/or CPUs to external programs, even though I'm not running any, etc).

I fully understand there is no 'one correct setting', as there are many factors with each system/project.... but considering my stats, and that I'm doing a standard def project (with lots of layers and pre-comps), can someone give me at least a general idea of what my settings probably should be at (...including any notes on things to watch out for, such as overscheduling, disk cache on/off, etc)?

BTW, I've read all the articles that are normally pointed to regarding RAM use, etc. Just looking for some advice for my particular setup, as well as an explanation on why it supposedly is only using 5% of my RAM (and if that can be boosted, or even NEEDS to be).

Thanks


Return to posts index

Walter Soyka
Re: RAM, CPUs and render times
on Sep 13, 2013 at 4:13:39 pm

My guess is that you are pushing everything to the limit, and multiprocessing is eventually failing due to a lack of RAM. When MP fails, it falls over to a single render instance.

You don't need to worry about filling RAM, you need to worry about shortest render time. Eventually something will bottleneck your render. You want that to be your CPU, since that's the most expensive and hardest-to-upgrade component in your system. What happens to CPU usage during these renders? If you look at the running processes, how many AfterFX.exe entries do you see?

It's counterintuitive, but sometimes lowering these settings will give you better performance. Throwing all your resources at Ae means there's nothing left for the system, and that leads to CPU overscheduling and RAM starvation. It's far better to throw only MOST of your resources at Ae so you can avoid these.

Leave a quarter of your RAM free for other applications (8 GB), and leave 2-4 CPUs available for other applications. Experiment a bit with your per-process memory allocation, but 3 GB is probably huge overkill for SD.

Walter Soyka
Principal & Designer at Keen Live
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
RenderBreak Blog - What I'm thinking when my workstation's thinking
Creative Cow Forum Host: Live & Stage Events


Return to posts index

Larry Adlon
Re: RAM, CPUs and render times
on Sep 13, 2013 at 6:12:50 pm

Hi, Walter. Thanks for the response.

Ya, it's kinda weird... It seems my machine isn't getting strained. The most RAM AE claims to be using is 8% (of the 32 gigs), and usually I have 3gigs per CPU set (although I've been experimenting with 2/CPU as well). At 1.5/CPU, it seems to crash, so I guess it needs 2gigs or more. At 2gigs, I get 11 CPUs, at 3gigs, I get 8. So, I'm assuming that I'm better off setting it at 2, since that gives me more CPUs, and the extra gig of RAM doesn't appear to be used anyway.

Normally, I have 3gigs for external apps, but I'm not running anything during the renders. But, by the looks of it, most of my RAM is just sitting being unused by AE (8%), so nothing seems to be gained from keeping it from external apps. But, at the same time, nothing is seemingly gained by having it reserved for externals since there are no externals running. All that expensive RAM, sitting doing nothing!

Still, I would assume that Windows itself needs something, as do the inevitable little programs secretly running under the hood on my machine... but I would assume that the 3gig minimum that I am forced to reserve for externals should be infinitely enough to cover them... I assume!

I forgot to mention that I have a CUDA card (NVidia GTX560 TI), which I'm assuming is being utilized somehow as well.

But, ya, even though I have a nice amount of power there, I worry that I may be bottlenecking or something as the result of TOO MUCH going in, as you are suggesting.

I'm assuming that (with my specs and with a standard def project) I'm better with the 11 CPU/2Gig route vs the 8 CPU/3Gig one.

I was also concerned about whether or not to have disk caching on, but my understanding is that AE will intellegently switch between disk vs RAM storage based on which it feels will be faster... so I keep it on.

What exactly is the difference between disk cache and the database?

I'm pulling frames from a fast SATA drive, running the O/S on a solid state drive, and rendering to that first SATA drive (...ya, not ideal to read/write on the same drive, but that's what I've got).

Certainly, the rendering is usually very fast, but sometimes it does add many hours to the renders for no obvious reason. Again, sometimes it appears purging memory does the trick. I imagine that maybe I should clear the disk cache when things go weird, too.

But, aside from the occassional 'WTF 30 hours?!' renders, I'm mostly just trying to optimize the power I have. It would be a shame if I have the power, yet am sabotaging the render times due to incorrect settings. I haven't been watching the CPU monitor, but checked them out during my last attempts. All 12 were pretty much at 100%, and I think RAM use was minimal. The AE background processors were all working fine (usually the expected 8 entries, including the front AE app).

If I'm not running any programs other than AE during the renders, should I still be setting aside more than the minimum 3gig default of RAM? Should I be setting aside a CPU for externals, if nothing is running other than AE? Do you agree that I should do the 11 CPU/2Gig configuration rather than the 8 CPU/3Gig? (1.5 doesn't seem to be enough, although I want to experiement with that again to be sure it wasn't just some random issue)


Return to posts index


Walter Soyka
Re: RAM, CPUs and render times
on Sep 13, 2013 at 6:50:09 pm

[Larry Adlon] "Ya, it's kinda weird... It seems my machine isn't getting strained. The most RAM AE claims to be using is 8% (of the 32 gigs), and usually I have 3gigs per CPU set (although I've been experimenting with 2/CPU as well). At 1.5/CPU, it seems to crash, so I guess it needs 2gigs or more. At 2gigs, I get 11 CPUs, at 3gigs, I get 8. So, I'm assuming that I'm better off setting it at 2, since that gives me more CPUs, and the extra gig of RAM doesn't appear to be used anyway."

You're measuring all the wrong things. You shouldn't care about how many CPUs you're using or how much RAM you are filling. You only care about how fast you render, and eliminating as many bottlenecks as possible.

Right now I am on a 12-core Z800 (24 with hyperthreading) with 48 GB of RAM. I've got 12 GB to other apps, and 12 CPUs to other apps. With this, I keep CPU utilization pretty consistently around 98% during render. That means the CPU is my bottleneck, and I can't make it go any faster on this system without changing the processors.


[Larry Adlon] "I forgot to mention that I have a CUDA card (NVidia GTX560 TI), which I'm assuming is being utilized somehow as well."

Are you using the ray-tracing renderer?


[Larry Adlon] "Normally, I have 3gigs for external apps, but I'm not running anything during the renders. But, by the looks of it, most of my RAM is just sitting being unused by AE (8%), so nothing seems to be gained from keeping it from external apps. But, at the same time, nothing is seemingly gained by having it reserved for externals since there are no externals running. All that expensive RAM, sitting doing nothing!"

Your comp may not be RAM intensive. Consider that while a single frame of 8-bit 1920x1080 HD uses almost 8 MB, a single frame of 8-bit 720x486 D1 NTSC SD uses only 1.3 MB.

There comes a point where more RAM will not help you. If you've reached that, but you're still asking Ae to claim more RAM anyway, you leave less free for the system and background tasks, and then they may require swapping.

In other words, you are chasing full RAM utilization, but that may not actually help you render faster. It may be doing the opposite.


[Larry Adlon] "I was also concerned about whether or not to have disk caching on, but my understanding is that AE will intellegently switch between disk vs RAM storage based on which it feels will be faster... so I keep it on."

Keep the cache on. If the reading from the cache is too slow, Ae will simply recalculate the frame from scratch.

You might consider moving the cache to your SSD if you have room.


[Larry Adlon] "I'm pulling frames from a fast SATA drive, running the O/S on a solid state drive, and rendering to that first SATA drive (...ya, not ideal to read/write on the same drive, but that's what I've got)."

Is storage your bottleneck? Are your CPUs sitting idle while disk access is through the roof during renders?


[Larry Adlon] "Certainly, the rendering is usually very fast, but sometimes it does add many hours to the renders for no obvious reason. Again, sometimes it appears purging memory does the trick. I imagine that maybe I should clear the disk cache when things go weird, too."

Again, it's my theory that multiprocessing is failing in these cases because you are being too aggressive with your memory settings. When multiprocessing fails, Ae reverts over to regular slow-poke mode, rendering one frame at a time with one process instead of rendering multiple frames at the same time with multiple processes.

Some details on your comp, the footage you're using, and the effects you're applying will help.

It would also help me help you figure out what's going on if you pressed Ctrl+Shift+Esc and watched the Processes tab for AfterFX.exe during render and reported back on the number of processes and their RAM usage, and if you watched the CPU usage in the Performance tab.

If you want to get really tricky, you can check out the disk queue lengths or or % highest active time for your storage in Resource Monitor.

Walter Soyka
Principal & Designer at Keen Live
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
RenderBreak Blog - What I'm thinking when my workstation's thinking
Creative Cow Forum Host: Live & Stage Events


Return to posts index

Larry Adlon
Re: RAM, CPUs and render times
on Sep 13, 2013 at 7:47:04 pm

Wow, some good info there! Thanks for your input!

I've been concerning myself about the CPU and RAM only because I assume they will aid in the speed of the renders (preventing the need to share memory, etc). But, ya, I know what you're saying. Really, it's not so much about 'my renders are slow', but more of 'am I at optimal settings?'. I'm trying to find the bottlenecks, if any.

No, I checked to make sure I didn't have ray tracing on for one of the comps. I thought that might have been one of the factors. That actually was the case one render when suddenly I was getting 'Your video card doesn't support the current configuration' error message. That was a moment of terror, right there, until I figured out the easy solution.

According to AE, the scene definitely is not RAM intensive (the 8% of 32gig display). I just kind of wish somehow I COULD be using it! Again, I'm not worried about taking RAM from the externals, since there's no externals running... but I totally see what you are saying there, too. It seems obvious that AE is not requiring it, yet I can't be certain how much Windows and any hidden apps/processes outside of AE are requiring... So, ya, probably better for me to put the 'unused' RAM there, just in case.

Although I've been starting to check the CPU readings in Task Manager, I'm going to have to sit down and really watch them, as well as some of the other readings, during the next render. I'm not fully knowledgable in some of their meanings, but I'm sure I can figure things out. I know the obvious things like CPU, RAM, etc... but am not fully clear on some of the other things, like Page Swaps, etc.

The project has about 130 visual layers (most are photoshop layers), with several of them being pre-comps. Everything is SD or smaller. Project is SD at 16bits. Effects range from standard AE filters (blurs, levels, etc), with some third party plugins (Optical Flares, Particular, and two Element3D layers). Most layers have expressions, mostly to randomize values. A number of Adjustment layers. Several layers are using modes like Add, Screen, etc.

I'll have to monitor the Task Manager carefully, and try and determine exactly what's going on. It's running fine, though. Just trying to optimize things, and sort out why it sometimes seems to take longer than usual.

Thanks a million for the detailed feedback. It's fantastic to see someone give full answers, and address each point made, rather than a blanket generic response. Really appreciate your efforts!


Return to posts index

Larry Adlon
Re: RAM, CPUs and render times
on Sep 13, 2013 at 8:33:02 pm

Just for your reading enjoyment, here's some of the readings I was getting from Task Manager and Resource Meter during a test render (Hopefully, I'm getting these right):

TASK MANAGER:
Overall CPU activity range: About 70-90
RAM usage (after it slowly ramped up): 23gigs
AfterFX.exe CPU mostly at 8, peaking at 32

RESOURCE METER:
The memory division bar:
Memory In Use: 23,000mb
Memory In Standby: 7150mb

Highest Active Time: Mostly 0 or 12

Everything seems to be chugging along nicely (visually at least). Not sure if the readings are good or not, but they seem to be (from what little I know). Frame times are generally 0-1 seconds, which is what they are normally when it's running fine.


Return to posts index


Walter Soyka
Re: RAM, CPUs and render times
on Sep 14, 2013 at 1:49:13 am

[Larry Adlon] "Thanks a million for the detailed feedback. It's fantastic to see someone give full answers, and address each point made, rather than a blanket generic response. Really appreciate your efforts!"

You're welcome!


[Larry Adlon] "Everything seems to be chugging along nicely (visually at least). Not sure if the readings are good or not, but they seem to be (from what little I know). Frame times are generally 0-1 seconds, which is what they are normally when it's running fine."

We might learn more if you tried next time you get a slow render to see what's different.

I tend to use Lloyd Alvarez's awesome BG Renderer script [link]. I haven't tested it scientifically, but I have noticed that it seems to peg my CPUs much more consistently than straight in-app renders.

Walter Soyka
Principal & Designer at Keen Live
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
RenderBreak Blog - What I'm thinking when my workstation's thinking
Creative Cow Forum Host: Live & Stage Events


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]