I just started having fun in new After Effect CS6 while doing Background replacement.
Found out I am not able to see the TrackMat option under the layer that I toggle switch.
Seems like it only appears in 2D layer, but not under 3D layer.
You're likely working with the Ray-tracing Renderer. If you do not need to extrude text or shape layers or bend video layers then you should be working with the Classic 3D Renderer - see the advanced tab in the comp settings window OR you can click the Render Type button at the top right of the Comp Panel to access the Advanced Panel.
Intensive AE & Mocha Training in Singapore and Malaysia
Adobe ACE/ACI (version 7) & Imagineer Systems Inc Approved Mocha Trainer