Okay, here is what I want to create; a button that is only ever visible when in the highlighted state, so the button jumps from item to item in the menu. My button design is a solid colour, but has a smooth curve, similar to the Nike swoosh. I've tried making a button where the swoosh is formatted with (=1) in Photoshop and nothing for the activated and normal states, however, when I build or simulate the menu the edges of my swoosh are very very jaggy and unevenly so. Is this a limitation of the subpicture format or am I doing something wrong? Is there a better way to do this?
It is a limitation of the subpicture format in DVD-Video. If you can, avoid using curves when designing subpicture layers.
If you must have the curves, then try also creating a stroke subpicture layer (with a (=2) prefix) and set its opacity to less than the original graphic subpicture opacity. Play around opacities a bit and it should make the jaggies less jarring.