If I remember correctly you needed to configure your audio card record options to use "What-U-Hear" mode. That was back in the days of Windows XP. I don't even know if it's possible anymore and I don't use Windows anymore (I'm on a Mac) so I can't check.
I believe that software packages like Camtasia can be configured to record system sounds because they are designed for doing demos in which this is important. Vegas Pro cannot, which is why you had to configure your audio card to do it for you.