According to Kush's guage (a formula for calculating adequate bit-rates,) Frame height x frame width x frame rate x motion scale variable(1,2, or 4) x Kush's constant variable(0.07). You get the number of bits per second required for a constant bit rate. For variable you take 3/4 of that number. That being said your target bitrate should be around 10.89Mbps. If you would like a better explanation for that number look at the link below.
http://www.paromitapoddar.com/VideoCompression/kush.html
Now you said you tried around that bitrate and it didn't fix the problem. the fact that you said that the file had this problem before sending it to vimeo means that it is not vimeo's display/compression settings. Now when I looked at the video I did see a couple small artifacts in the color especially in the papyrus colors, but I did not see which choppiness you were referring to exactly. Was it the aliasing you were referring to? because there is quite a bit of that, but that is naturally going to happen on a h.264 compression of such tight small lines.
Doyle Lewis, Assistant Videographer
thinkck.com
