I've made a 77 minute documentary in final cut pro 7. The footage was from a variety of cameras: sony z1u (1080i60), sony EX1, sanyo xacti (h.264), sd footage (mpeg). We used after effects cs3 for different animations (title graphics mainly). I went back and converted to pro-res all the misc formats.
My process from timeline to dvd: export as a quicktime reference file (hdv), go to Compressor where I selected "DVD best quality 90 minutes". I then took that file to Encore CS3, where we had a video background menu, made in After Effects.
AFter burning the dvd, I played the dvd on a 32" LCD TV. The video menu looks pretty good, but the 77 min video is not good. The text is distorted (readable, but visibly poor quality). The slow motion parts of the video were very stuttery. The skin tones of people are more red. The detail is not there. I'm not looking for high def, but it's just not good compared to the other sd documentaries I've watched.
I have also tried the method, of dropping the hd timeline into an sd timeline in fcp, and then exporting an sd quicktime reference file > Compressor > DVD best quality 90 minutes > encore cs3. I got the same results.
I'd appreciate any help for getting the best quality end product on a DVD.
Just a guess, but the problem may be Encore. If you are using CS3...heaven help u:) And I don't think it improved much in CS5 either, although adobe media encoder did. I have no problems doing what you describe only I use DVD Studio Pro (as part of FCS3). It should not take you long to regen the menus there, if the DVDPS DVD encoder works better for ya. Good luck!
Phoenix Studios Nepal: A small A/V Production House in Kathmandu.