On a decent NVIDIA card, the ray-traced renderer should be somewhat decent.
What version of AE are you using?
What are your ray-traced settings for the GPU? Do you have "use unsupported cards" checked?
If you are on CC 2017.1 and that box is checked, you may need to swap out the Optix library on your machine.
Rename the OptiX.1.dll from the After Effects program directory to Optix_old.1.dll (that way you can put it back to normal if you need to). This is typically the C:\Program Files\Adobe\Adobe After Effects CC 2017\Support Files directory.
Then install the latest version of OptiX from the NVIDIA website (https://developer.nvidia.com/optix ).
After doing these two steps, the ray tracing functionality in After Effects should be functional.
As an aside, after removal of the OptiX dll from the After Effects directory the CPU ray tracing support will not work, but that was a pointless feature anyway, so I wouldn't worry about it.
- The Great Szalam
(The 'Great' stands for 'Not So Great, in fact, Extremely Humble')
No trees were harmed in the creation of this message, but several thousand electrons were mildly inconvenienced.