I'm trying to do something which I originally didn't think it would be too difficult but it definitely is. What I have is a scene with a table and wine glass and I want to have a particle system travel behind the glass and then wrap around in front of it. What I originally did was motion track the glass with the Foundry's AE Cameratracker, then placed a solid where the glass should be in Z space and wrapped the particle system around it. I couldn't get it to be obscured by the solid though, unless I just placed the layer on top of it. I've got a whole different problem after that which would be to have the particle system perhaps blurred out behind the glass and then not in front of it. Track matte's won't work..
While typing this I'm thinking that maybe cutting out the glass and offsetting in in 3d space would work?
This project isn't important, but now that I've come across this problem I'm really curious to figure out a solution.
Things will depend on which particle system you're using. Particular, for example, allows you to choose a 3D 'obscuration layer' which will hide particles passing behind it. You need to be careful though as particles can drift in 3D space and occasionally pop through the layer, but if your particles are following a motion path, that may not be a problem.
The other solution - or workaround - is to duplicate the Particular layer, and sandwich your 3D glass layer in between the two. You can then use the z-buffer settings in Particular to show only particles in front of, or behind, the glass on the appropriate layer. You'll probably only want to do this once you've locked down your particle effect, to avoid having to redo the changes, although you can use expressions to tie the Particular settings on the two layers together, and I believe there is also a 3rd party script that can create the duplicate layers for you.
Another way of doing this is to use FreeForm Pro. You can create a displaced bottle, position it in 3d space and render out a distance map that can be used with Particular to hide particles- not the most straightforward though.