Put your layer with the "silhouette alpha" on top of everything and then place an adjustment layer below it. Don't apply any effect to this adjustment layer. The adjustment layer placed below 3d layer will cause the 3d layer above it to ignore the 3d space and use order of layers.
What Bartek said will make it clip everything in that 3d layer's area. To clip only what's in front of it:
duplicate your main composition(in the project panel), then:
1- tint the 3d clipping layer white, and the 3d comp black.
2- Put these two comps (your main comp, and the tinted one) in another comp.
3- Set the main comp to use luma inverted of the tinted comp.
4- Precompose that (let's call this Matte1). Bring in the 3d layer. Set Matte1 to use the 3d layer alpha's matte.
5- Precompose this and use it again as an inverted alpha matte to your initial 3d comp.
Guess it will work, if I got something wrong, maybe this can at least give you some direction.
Can you clarify your directions? I've tried following your steps, but I'm having some difficulty.
I have a very simple composition.
I have text animating on a circular mask with the center at 0,0,0.
I have no problem setting up my " Z-plane" and making the positive Z space (or further from my camera) masked out.
I used a square comp with white solid and placed it in 3D space at 0,0,0.
It's above the text path in the composition and set it to silhouette alpha.
The obvious challenge is making the positive (or closer to the camera) masked out.
Let me know if you have any ideas,
I'm sorry but I don't know how to be more clear, maybe my instructions are hard to get at first, but they are all there. What version of AE do you run? maybe then I can create a simple project to illustrate.
The goal of what I was saying is that... if you can mask what's behind something. You can use THIS as a matte to get the opposite.