I agree that Sony AVC will give you a good encode. Your problem may just be your bitrate. If your camera shot 16Mbps HD and your rendering 5Mbps WMV that's 1/3 the information. Sony AVC with 16Mbps should produce the same quality as your source. Even at 8Mbps it should still be pretty good.
~jr
http://www.johnrofrano.com
http://www.vasst.com