Chris,
I’m not sure which anti-aliasing settings you are referring to in Smoke so if you can elaborate that might help. But you might try adding a edge tool after the mask and turn on shrink. This will obviously shrink the mask a little but it also generates a smoother edge.
Ken LaRue
Autodesk Application Engineer
Maya Composite / Toxik / Flame / Combustion
Training DVDs:
http://www.thestreetproductions.com