I have created subtitles in a utility called Belle Nuit Subtitler, that exports them as tiff files with alpha. I exported a FAB text file, and then I imported the this FAB file and the subtitles to Encore.
When I play the timeline, the subtitles play with black background and the characters are not antialiased, they appear terrible.
How can I get antialiased subtitles with Encore DVD?
Any claims of "anti-aliasing" in the DVD-Video format is a bit of an exaggeration, compared to what one expects of anti-aliasing of text or vectors in other computer graphics applications. In DVD-Video, subtitles exist in a 2 bit graphical overlay. With only 2 bits, there's not a lot of anti-aliasing that can be done. You basically get color and opacity plus overall transparency in this graphical layer.
You'll never see anti-aliasing like you would from anti-aliased text in Photoshop, After Effects, or Premiere.
Can you post a screenshot of how your subtitles look? I could tell you whether they looked worse than normal, and perhaps make some corrective recommendations from there.