All of my YouTube (and Vimeo) videos are rendered using the MainConcept HD for Internet 1080 render choice, and I turn up the average and maximum bit rates. If I need the utmost quality, giving up a faster render, I enable two-pass rendering.
What you have not provided to us is how you are capturing the movie footage. I know the capture software my son uses has a pretty low capture nitrate, which yields a poor quality. If I were you I'd research what method is best for your capture needs.
I've been using MagicDVDRipper on standard discs for the most part. I use a higher bitrate on that to get better quality footage than I was previously getting. Recently I ripped in AVI as high as I could get.