This problem with Encore has been annoying me for a while.
Basically when we're playing the main feature timeline - or any timeline for that matter, upon pressing Next Chapter or Last Chapter on the last or first chapter points the DVD goes into a random DVD menu.
I understand what's happening, the DVD players are reading the titles listing and moving on as they are in sequence, but on a PC DVD player the correct action of going back to the main menu happens.
There's no way to fix this. As far as I know, there's no definition in the DVD spec of what to do when the Next Chapter button is pressed, but there are no chapters left in the timeline (or Previous when you're on the first). So it's left up to the player to decide what to do. Set-top players tend to pick the next timeline, and apparently your software player commits the end action. Some players I've seen don't do anything.