Often times deinterlacing issues can be content specific. While Compressor's deinterlacing can be very good, the controls are extremely limited. If you post an example encode (moving video, not still as that's important) maybe we can diagnose better.
You might have to look at something like Telestream Episode which has very deep deinterlacing controls. You may need something like Edge Detection or Motion Compensation (what Compressors does at its best setting I believe) or Deinterlacing Moving Areas or even create a manual threshold.
[Gareth Tunley]"The original Hi-Res source file is DVPAL and has no interlacing artifacts."
This is a fuzzy comment. Unless your DV PAL is Progressive it is Interlaced. I'm not sure what "artifact" would mean in that context. It either is or it isn't and if it is . . . it is. That you see it or not may be how the file is monitored/played back.
Thanks Craig. Sorry I missed this reply when you posted it. Very helpful.
I will definitely investigate Telestream.
I can see my comment was indeed "fuzzy" so apologies for that. I still find the language of interlacing/deinterlacing confusing! What I meant was that the original footage/fcp viewer showed none of the jagged lines but after attempting to make it to H264 these horrible jagged lines started appearing during motion sequences. The original footage was shot on a Z1 so it would be interlaced I assume, but in the "good" way (that is without the jagged lines). I suppose I'm still confused as to why these lines appear when changing codec...
The advanced edge detection and motion compensation options in compressor brought in estimates of 16 hours plus for the compression which wasn't really an option on the deadline I was on.
For what it's worth: I eventually solved the issue by compressing in MPEG streamclip. I don't understand why but it seemed to work.