Subpicture highlights are only 2-bit colors plus transparency. That limits you to a solid color shape. You can combine up to 3 solid-color shapes for a single button highlight, but that's as far as you can go.
Images, mask feathers, effects, etc. from Photoshop can't be used for the highlights directly.
To get around this, you can create duplicate menus that are activated automatically when the audience navigates to a button on the original menu. The duplicate menu can contain the image that you wanted to use as the highlight in place of the button on the original menu.