I am trying to figure out a way to get better quality results on encoding mpeg2's that have a lot of the color red in them. First I will describe my project. From the Red Camera we took and edited footage that was converted in red cine to Avid DNXHD 1080 and cut in an avid. Then the final sequence was output as a quicktime source to a Digital Rapids Stream Z HD box and transcoded to a high quality mpeg 2 elementary stream for authoring. Once burned out from dvd studio pro the red channel has very apparent stair stepping to it. I believe this to be because of the 4:2:0 color conversion needed for the dvd specifications. Is there any way around this? And just incase some people are thinking it is the down conversion form HD to SD (which is normally a culprit for a lot of people) This problem still occurs on native SD footage. It happens whether I use the Stream Z box, compressor, media press X hardware, avid mpeg2, no matter what is used for the encode I still have the problem. Thoughts?
This has always been an issue with the mpeg2 codec, even noticeable in hollywood dvds. The stair stepping can be reduced with very advanced algorithms, but it cannot be eliminated. You can watch the Disney movie "Cars" for example. They did a very good job of reducing it, but it is still noticeable. You might want to try using AVISynth (for downscaling) and HC Encoder to encode to mpeg2. HC Encoder allows for custom matrices which will allow you to reduce it if you know what you are doing. It will require lots of research though. BTW, those programs are to be used on a PC, and are completely free.
I wrote an article on using those programs in a HD to SD DVD downscaling workflow: