BCC 6 for AE (Win) OpenGL filters are shown as 16bit, but only seem to work in 8-bit (in 16-bit I get just a black screen or AE rendering error). The BCC 6 release notes mention Rays as not supported on 16-bit (but they 'do' seem to work), but no mention of OpenGL. Oddly, the release notes for BCC 5 do note that the OpenGL filters are not supported for 16-bit. Clarifications, if you would?
The various Boris products (Red, Blue, FX, etc.) all show up as 8-bit only filters in AE (CS3), but everything I've seen suggests they should be seen as 16-bit. Are they only 8-bit supported?
What video card/memory amount do you have on your AE system? The only explanation I can offer is that the system is running out of OGL resources. Unfortunately, this is a fact of life with OGL rendering. You need a beefy card.
You are right in your observation that in BCC5 OGL filters were limited to 8 bit. That was one of the BCC6 improvements.
RED 4, FX 9 and GR 5 should show up as 16 bit in AE. Earlier versions may be 8 bit only.