I've noticed many of the same symptoms. Nucleo did speed up my XP64 4-core Opteron machine last year with little effort on my part, but the 8-core Mac Pro using AE CS3 and Leopard took some tweaking to make efficient.
I have an 8-core Mac w/ 8GB RAM. I am testing a file I made on a mac mini running AE 6.5 (no nucleo). It took 2 hours to render on the mac mini, 32:54s to render on the Mac Pro without Nucleo and no multiprocessing and 33:18 to render WITH multiprocessing checked (8 memory starved cores = slower render).
My Nucleo tests have shown that with 8 cores, a Nucleo render is twice as fast as AE with multiprocessing no matter how I split the processing. (my comp is layers of video, 3D animation, nested .ai comps and other images -- probably 400-500 layers, 2min long,
link to the animation)
Next I went through the Resource Settings in Nucleo and did a test render for 4, 5, 6, 7, and 8 cores enabled.
4 cores - 13:04s
5 cores - 12:31s
6 cores - 11:21s!!!
7 cores - 12:24s
8 cores - 13:10s
All of these tests were with OpenGL turned on and all were about a minute slower with OpenGL turned off. Likewise, all of these tests were with Nucleo on normal performance setting and they all were about a minute slower with Max performance on (strangely).
I also had an issue with blank (all white) renders using Nucleo. It was because I used an alternate color management setting under the project settings menu. Seems Nucleo will only render with this setting set to "none" (default).
Another unresolved thing is that my nested .ai comps rendered perfectly with Nucleo, but the AE renders have produced very blury edges -- haven't had time to investigate that though.
So yeah. I cut a 2 hour render to 11min with a couple Xeons and Nucleo... 6 cores seems to be the magic # for me until I can afford to fill up the other 4 ram slots.
Hopefully you've had enough time to acquaint yourself with all the parameters within... And hopefully this will resolve your blank renders issue too.
_E