I didnt see a need for an alpha channel (you mentioned millions of colors plus) but it won't be whats causing the stuttering.
I didn't see any in the clip you posted. If it's a banding or interlacing thing, you'll find many avid codecs not happy in other applications.
Do you need tc? Have you tried just capturing directly with deck control turned off? Just trying to skip the encosing step to see if thats the culprit.