The problem (of not directly supporting rollover type buttons) is not caused by the authoring software, but by the DVD specification which doesn't offer any other tool than simple subpicture highlighting. This means that anything else can only be accomplished via workarounds.
For an Encore workaround have a look at the Adobe Encore forum FAQ:
How do I create a Button Rollover effect?.
DVD Studio Pro basically does the same. The difference is that in Encore you have to do the whole workaround by hand, while DVDSP does it all for you from a layered .psd file. Ultimately both the Encore and the DVDSP authored DVDs will have a whole stack of menus filled with auto activated buttons - with the associated drawbacks. (Sluggish navigation and no mouse navigation on software DVD players.)
A real solution can only come from a new DVD spec and a new generation of set top DVD players, as the current generation of players wouldn't support such a change.