When you tried what I suggested, did you change the particle layer to be an alpha matte with the mask layer directly above it? From your description it seems as though you may have set the actual mask layer to alpha matte, which certainly wont work.
As for your other idea yes it should work but I would try a precomp of your particle effect first. Select your particle layer (and any other layers if the particle effect is using them) Next click, Layer > pre-compose. Choose "move all attributes into the new composition" and click ok. Now you will be able to apply any mask you like to the new precomp layer sitting in place of your particle layer.
3D animator and VFX specialist