1.Create your transition (After Effects, Motion)
2.Render to DVD compliant MPEG2 stream (m2v and wav)
3.Add track in DVDSP
4.button action to goto transition track
5.transition track end action to original button destination.
since transitions are usually small you can duplicate the transition track for how ever many buttons you have on your DVD menu. Each ones end action going to your button destination.