Dude, it ain't the codec. It's the way you are manipulating it after you digitize.
DV video recorded via FW is a data transfer and the codec process is absolutely not involved. The Apple QT is merely a wrapper. Secondly, when you capture in a non-native codec such as SDI 4:2:2 10-bit via Sony, the DV data on tape gets encoded from 4:1:1 to 4:2:2, and 8-bit gets re-quantized to 10-bit (the only system that allows the Sony SDI 4:2:2 codec to be read as a pure data transfer from tape is the Sony Vegas. Sony does not share their codec with FCP like Panasonic does). This is an impure process, though the results could still be fine looking. It's just not what is actually on tape -- see below (goto 'which codec is better'):
http://www.adamwilt.com/DV-FAQ-editing.html#codecs
See test #2 comparison and conclusion at bottom:
http://www.nattress.com/Chroma_Investigation/chromasampling.htm
Again, the proof is in the results as you see it. However, everything I have read on this issue points to two conclusions, AFAIC: 1) the difference appears to be inconsequential enough unless you are doing multi-generational graphix, 2) the disk drive space trade-off is generally not worth the expenditure.
It could be (though unlikely) that you have a lousy FW cable that could be introducing artifacts, or that the DV transfer is actually showing you MORE than the SDI transfer, which accounts for the extra crap you see. Although I have never bought it, I have read that there are special 4:1:1 8-bit DV specific chroma key plugins that supposedly do a darn good job, for DV. Maybe the 'stock' keyer just isn't cutting it? Found this a moment ago:
http://www.dvgarage.com/prod/prod.php?prod=dvmattefcp
steve covello
double wide post