DVD bad quality
When I create a new project in Encore CS3, the only frame size available is 720x480. All other options can not be changed as well, except for NTSC/PAL. Using these settings I created a test project. All elements are created inside photoshop as vectors, and solids with effects in after effects. According to adobe's help files, this is the best way to preserve quality when the movie is scaled up. But when I build the project (which is only a menu with animated video as background), the quality is horrible. Can anyone help please.
I haven't used Encore yet but I may be able to offer some advice. I normally use Apple's DVD Studio Pro for authoring, and as such have developed a workflow of creating two sets of assets for menus: one for editing and one for authoring. Since DVDSP doesn't support layer effects from Photoshop, I usually flatten the image to a .tiff or .png first before using it as a background. Or, if I am using a motion background I will create all my motion graphics in Final Cut Pro or Motion and them render them using Compressor to .m2v files before importing them into DVDSP. That is, I try to do as much design work outside of DVDSP as possible and only import assets that are already compressed and ready to burn.
Using this same kind of method may help improve your quality using Encore. This is just a guess but I think the reason you are seeing poor quality has something to do with the way Encore is encoding your assets. Instead, I would recommend compressing your menu assets outside of Encore first, using whatever program you think produces the best results, and make sure Encore is not re-compressing when you build the DVD.
Another thing I do, which I don't know yet if Encore will support or not, is build all of my buttons outside of DVDSP and create overlay files for the highlights. This is essentially an image file that is simply pure black and pure white, with the black areas defining where a highlight will occur. Then, after setting that image as my overlay file in DVDSP, I create my buttons without assets, so they are just defining an area and not adding anything visual to the menu. Anything within the area of a button that is black on the overlay file will show up as a highlight, so you can define several button highlights with only one overlay file.