FORUMS: list search recent posts

Quality issues with animation using Compressor in creation of SD DVD

COW Forums : Compression Techniques

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Linda Chirrey
Quality issues with animation using Compressor in creation of SD DVD
on Mar 14, 2011 at 4:28:05 pm

Hi,

I am trying to create an SD DVD of a film project that contains a mix of live footage and animation (inter-cut, not in a Roger Rabbit way). The quality of the SD DVD, when watched on an HD TV (1080p) is really pretty poor, but not uniformly for the different footage. The film will be mastered in HD and the DVDs are just for showing work in progress, but ultimately I want to produce festival screeners etc, so need the quality to be vastly better. I’m hoping that someone might be able to suggest where the problem is and what can be done to improve the overall result.

For the animation, when there is little or no camera movement, the main problem is jagged lines on the characters’ black outlines and a bit of noise. Pans and pull backs cause significant noise and a pronounced shimmering, especially on finer lines. The live footage is ok-ish (darker, with a loss of vibrancy in the colours). Text graphics, particularly when placed over live footage are poor (very jagged). Simple white text on black background is ok.

My workflow is as follows:

Hardware for all processes:
Macbook Pro 2.6 GHz with Intel Core 2 Duo,
OS Mac OS X v 10.5.8.
Memory: 4GB 667 MHz

Animation produced start to finish on Toon Boom Studio 5.0 (Build 5.0.13834).
Live footage shot on XDCAM EX at 1080p25
Editing is on FCP (v7.0.3). Text graphics are generated on FCP. The FCP file is exported directly to Compressor 3.5.3 to create the MPEG2 and then brought into DVD Studio Pro (v 4.2.2) to create the SD DVD. SD video format is PAL.

The TBS settings are:
Aspect ratio: 16 x 9
Resolution: 1920 x 1080
Frame rate: 25 fps
TBS Export settings: to QuickTime movie
Compression Type: Apple Pro Res 422 (HQ); Quality High; Automatic Gamma Correction; Progressive; Dimensions 1920 x 1080; frame rate current
All filter settings as per default

The timeline settings for FCP are as follows:
Sequence settings: XDCAM EX 1080p25
General: Frame size 1920 x 1080; Aspect ratio HDTV 1080i 16:9; Pixel Aspect Ratio ‘square’; Field dominance ‘none’
Video Processing: Render all YUV material in high precision YUV; Process Maximum White as ‘white’; Motion Filtering Quality ‘best’
Render Control: Filters & frame blending for speed – both checked
Render: frame rate 100%; Resolution 100%; Codec Apple ProRes 422 (HDV, XDCAM HD/EX only)
Master Templates and Motion Projects: best; ‘Always use Best when Rendering Movies’ – checked.
I’ve pretty much tried all variations of encoder settings in Compressor for DVD Best Quality – presets and tweaking the various video format options (e.g. field dominance, quality settings, GOP and frame control and filter options). None of the variations made any observable difference. The only thing that improved the animation slightly was applying a noise removal filter in the chroma channel at 4 iterations. Although it reduced the noise, it didn’t improve jagging significantly of animation or text and made the rest of the footage soft.

If someone out there is routinely doing this and can recommend some appropriate settings and where the problem may be coming from, I’d be really grateful to hear from you!


Return to posts index

Jeff Greenberg
Re: Quality issues with animation using Compressor in creation of SD DVD
on Mar 15, 2011 at 1:19:45 am

[Linda Chirrey] "If someone out there is routinely doing this and can recommend some appropriate settings"
Not with your exact footage.

I think what's killing you is the export. You're either exporting via ProRes (decent) or XDCAM.

Some thoughts:
First, don't export. Use 'send to.' Exporting is forcing a render (first) and compression and THEN it goes to SD DVD. Just use the send to compressor choice.
Second: why not take 30-40 sec of the animation and process it (rather than the whole piece.) If you get that working...everyrthing else is secondary.
Third: go out of TBS as ProRes4444 or animation; both will produce giant files (that dont' play so great) but will be closer to uncompressed.

Keep logs of what you do. Try those three. Also, tell us what settings you're using in Compressor. One thing that will make a difference (but make the compression MUCH longer) is to change the compressor frame controls to improve the downconversion. Turn them on, change the resize to 'best' and the deinterlace to "Best". Increase the anti-alias and the Details level.

By running 30 sec through (multiple times, multiple setups and settings) you can preview the results for much longer files.

Best,

Jeff G

Apple Master Trainer | Avid Cert. Instructor DS/MC
Come See me speak at NAB!
Compressor Essentials from Lynda.com
(older but still good) Marquee, Media Composer (3.5) and Basic/Advanced Color DVDs (1.0) from Vasst.com
Contact me through my Website


Return to posts index

Linda Chirrey
Re: Quality issues with animation using Compressor in creation of SD DVD
on Mar 15, 2011 at 4:24:55 pm

Hi Jeff,

Thanks for getting back to me.

To clarify, I’m using ‘send to’ rather than exporting to Compressor from FCP.

Take you point about concentrating on the animation and getting it right in the first place.

I’ve now tried as you suggested exporting from TBS using Animation and Apple ProRes4444. For some reason I couldn’t export the ProRes 4444 from TBS (kept getting an error message saying ‘unable to export, might be in use with another application’). I tried 10 bit uncompressed instead. Same process as before, bring into FCP timeline then sending to Compressor. Unfortunately the export using Apple ProRes 442 is still looking the best of the bad lot.

Re the Compressor settings, testing without any filters etc, the best I have found so far (and the ones I checked the above exports with) are as follows:

Video Format: PAL; frame rate - 25; Aspect Ratio -16:9; Field Dominance - Progressive
Quality: mode – 2 pass VBR best; Av bit rate – 5.0Mps; Max Bit Rate 7.5Mps
GOP: Structure IBBP; Closed; GOP size 12

Frame Controls: Frame Controls – on
Resizing Control: Resize Filter – better (linear prediction)
Output Fields: Progressive
Delinterlace: Better (adaptive motion)
Adaptive Details – checked
No anti-aliasing or details level applied at this stage.
No filters applied

With regards to the Compressor settings, I’ve tried all the possible variations for Frame Control (i.e. the resizing control, output fields, deinterlace & checking/ un-checking adaptive details).

I’m using a mid-range av. bit rate (5.0) as the film is quite long (circa 120 mins) and needs to fit on the DVD.

If you can shed any more light, that would be really helpful.

Many thanks,

Linda


Return to posts index


Jeff Greenberg
Re: Quality issues with animation using Compressor in creation of SD DVD
on Mar 15, 2011 at 5:05:44 pm

If it looks great in FCP, but lousy on a DVD, then the problem is that part of the chain. No point in examining anything else (as far as my thought process goes.)

I'll in parenthesis explain each of my thoughts.

We need to examine where something is going wrong:
Between FCP & Compressor - compressor has had a history of bugs, more towards PAL than NTSC frame rates.
Between the compressed video and DVDSP
Between DVDSP and the end viewing - the TV set.
(you have defined a problem, we have to figure out where it actually exists.)


You don't say where you're evaluating the problem. You shouldn't truly evaluate an MPEG 2 on anything less than the end product - the type of 1080p screen you're looking at.
(it might look bad on your computer, but fine in the final playback mechanism.)

And there's a real quagmire in the SD on an HD set….you're downconverting your work to view; and having the TV set do the upconvert - no matter what you do, it might just not look good being blown up. Can you grab a DVD of other animation and see if it suffers similar artifacts.)
(In which case, its the upscaling and there's nothing you can do about it.)


Other general thoughts;

If you're not planning to put chapters, try setting the GOP to open. This will permit more variance in the VBR.
(if this works, then that's the solution)

I'd use a 30 second compression out of compressor and set it to a huge bitrate. The whole idea of only 30 seconds, is that you can also test the adjustment for antialiasitng and details. Try one with them set to nothing; one with them both on, one with value high, and the last with the other.
(by doing 30 second moments, one of them may 'hit' and solve the problem)

You need to take the bit rate out of the picture as a factor (or expose it as the problem) - you have to boost the bit rate to the maximum (around 7.5 mb/s) just to make sure that a larger/higher quality wouldn't' solve the problem.
(increasing the data rate removes the lower data rate as a factor)

As long as we're going down this route…..why not put out a 1pass VBR and a CBR, just to check.
(Using 1pass and CBR may solve this, if there's a bug in comrpessor)

Last, I'd download a copy of bitvice and try it to see if that improves everything.
(Let's take compressor out of the picture - bitvice has a demo.)

Then put them all on a DVD and actually check them on the final product - the 1080p tv.
(again, where are you monitoring?)

Last, if you want to post 30 second somewhere, I'd be happy to push it through some of my encoders here (Squeeze 7, Episode, bitvice) to see if any of them have the problem.

Best,

Jeff G

Apple Master Trainer | Avid Cert. Instructor DS/MC
Come See me speak at NAB!
Compressor Essentials from Lynda.com
(older but still good) Marquee, Media Composer (3.5) and Basic/Advanced Color DVDs (1.0) from Vasst.com
Contact me through my Website


Return to posts index

Linda Chirrey
Re: Quality issues with animation using Compressor in creation of SD DVD
on Mar 16, 2011 at 8:11:07 pm

Hi Jeff,

Many thanks for your comments and suggestions.

To clarify, we are examining the problem on a 1080p TV, hooked up to a Pioneer DVD player. We did wonder whether it was an issue with the HD TV handling the SD DVD, but hadn't reached a conclusion. When we play the DVD on the Mac, viewed on a computer monitor (a Dell UltraSharp 2408WFP) the issues do not show up to anywhere near the same degree – in fact it is almost acceptable. Similarly, when we hooked the Mac up to the TV via a HDMI cable we get pretty much the same results as viewing it on the Dell. Not bad at all.

As per your suggestion we watched a couple of animation DVDs on the DVD player/ 1080p TV. Both are commercially produced DVDs. Although the older one (Mr Magoo) did show some noise, the more recent one (Spirited Away) looked really fine – certainly none of the issues we are seeing here.

If you can unpick what all of that means...

I ran the various tests you suggested versus a control with the same settings as described yesterday, using a 30s animation only clip. Kept all settings as per described yesterday, changing only the one noted below:

Setting 1 – High Bit Rate (7.5Mps)
Setting 2 – GOP open
Setting 3 – 1 pass VBR best
Setting 4 – 1 pass VBR
Setting 5 – CBR
Setting 6 – anti-alias 0, detail 0
Setting 7 – anti-alias 75, detail 0
Setting 8 – anti-alias 0, detail 75
Setting 9 – anti-alias 75, detail 75 (failed test, see results below)

Settings 2 – 9 were tested using both the standard av. bit rate 5.0 and high bit rate of 7.5.

Results
Setting 1 - Changing the bit rate improved the depth of the colour slightly, making it a bit richer, but other than that, there was no discernible improvement with the noise, shimmer and jagging.
Settings 2 – 7 - None of these changes showed any noticeable difference, whether at std or higher bit rate.
Setting 8 - anti-alias 0, detail 75 - this was really horrible as it accentuated all the issues and washed out the colour.
Setting 9 – anti-alias 75, detail 75 – this setting could not be run at all. A failure message was repeatedly displayed. I tried dropping the values as low as 10 each, but still no joy.

We have downloaded the trial BitVice and will give it a try tomorrow. Hopefully it will give us a better result!

If not, I may take you up on your kind offer to pop the clip through a couple of other encoders to see what happens.

Really appreciate all your help on this and look forward to your comments.

All the best,

Linda


Return to posts index

Jeff Greenberg
Re: Quality issues with animation using Compressor in creation of SD DVD
on Mar 17, 2011 at 12:04:49 am

I'm enjoying this puzzle (sorry, I'm sure you're not!)

Hey, as long as your running tests, how about generating an uncompressed QuickTime out TBS at 720x480 (not 486) and then punching that through compressor (let's take FCP out of the picture and see what happens.)

Totally willing to throw the video through my toolset here.

Best,

Jeff G

Apple Master Trainer | Avid Cert. Instructor DS/MC
Come See me speak at NAB!
Compressor Essentials from Lynda.com
(older but still good) Marquee, Media Composer (3.5) and Basic/Advanced Color DVDs (1.0) from Vasst.com
Contact me through my Website


Return to posts index


Rich Rubasch
Re: Quality issues with animation using Compressor in creation of SD DVD
on Mar 17, 2011 at 1:05:42 am

Simplest way to make a great looking DVD from an HD timeline in FCP is to Export a test clip (as Jeff suggested) as Quicktime ProRes 422 (does NOT need to be HQ) and scale it to 720 x 480 widescreen. If you are in a progressive timeline in FCP then you don't need to specify fields in the ProRes advanced box.

Bring that into compressor and I'll bet that if you even use 6.8 mbps CBR it will still look great.

It's the only way we get rid of the jaggies.

Try it and report back. Remember just a test clip with some of the more challenging clips.

Rich Rubasch
Tilt Media Inc.
Video Production, Post, Studio Sound Stage
Founder/President/Editor/Designer/Animator
http://www.tiltmedia.com


Return to posts index

Linda Chirrey
Re: Quality issues with animation using Compressor in creation of SD DVD
on Mar 17, 2011 at 8:02:45 pm

Hi Rich,

Thanks for taking the time to reply. We had previously looked into option of making a Quicktime from FCP and bringing that into Compressor, but as the results we were obtaining were routinely worse than sending to Compressor from FCP, we had pretty much eliminated this option.

Anyway, in the spirit of always being prepared to double check, I thought I’d give it another go by following your suggestion.

I’m UK based, so I went for the PAL frame size of 720 x 576 instead of the suggested NTSC 720 x 480. I tried Quicktime Pro Res PAL anamorphic, Pro Res PAL, as well as custom settings as 720 x 576 (frame size) CCIR 601/DV PAL (5:4)(aspect ratio). Compressor was set to 6.8Mbps and CBR, with all the other settings as previously described.

Unfortunately the results were really bad – much worse than we have been getting by using ‘send to’. Very pixely, loads of noise with very washed out colours.

We did try making a 720 x 480 using Pro Res 422 NTSC Anamorphic, out of curiosity as we are using a multi-standard DVD player, but obviously it didn’t like being played on the TV and looked even worse than the PAL settings.

Maybe the results would have been improved by using a two pass VBR, but because the results were much as I had previously obtained by creating a QT first, I didn’t think it was really worth it at this stage.

We have been working our way through some more tests, as suggested by Jeff, so if any of that info plus the above gives you any ideas of what else to try, I’m all ears!!

Thanks and all the best,

Linda


Return to posts index

Linda Chirrey
Re: Quality issues with animation using Compressor in creation of SD DVD
on Mar 17, 2011 at 8:07:26 pm

Hi Jeff,

I don’t mind a puzzle, as long as there is a nice answer at the end that’ll put me out of my misery!!

I tried creating an uncompressed QT from TBS. I used 3 variations: compression ‘none’, uncompressed 10-bit and uncompressed 8-bit. Rather than using the 720 x 480 NTSC dimensions I set it at PAL 720 x 576, 16:9.

I did two variations for each in Compressor: one with field dominance set as ‘upper first’ the other as ‘progressive’.

As for the QT created from FCP and reported in my reply to Rich, the results were also really bad. Very pixely/jaggy, lots of noise, shimmer. Much worse than we have been getting. There was not obvious difference between the field dominance variations.

We have started to do a bit of work with BitVice. The results using the default settings were terrible. However (and perhaps a glimmer of hope), when we unchecked the interlaced box, there was a noticeable improvement in the jagging/pixilation. It didn’t eliminate it completely, but definitely an improvement. Without any filters applied, the noise is worse than we’d been getting. However, upping the bit rate and applying Spatio at ‘blur tougher’ cancelled a lot of the noise, to the extent that we are probably now achieving equivalent results with BitVice as Compressor.

Other than being more straightforward, the other thing that is more obvious with BitVice is that different settings really do give different results. With our Compressor, you’d be hard pushed to see any difference between any of the settings. In your experience, is this normal?

Anyway, still some combinations to play with in BitVice.

In the meantime, we’ve uploaded the 30s clip of animated material we’re working with to Dropbox. I’ll email you the link and if you have the time and inclination, you can get 'hands on' with the puzzle!!

Thanks again and look forward to hearing from you soon.

All the best,

Linda


Return to posts index


Linda Chirrey
Re: Quality issues with animation using Compressor in creation of SD DVD
on Mar 21, 2011 at 9:18:37 pm

Hi Jeff,

Many thanks for sending through the various MPEG2s etc. We burned them on a DVD using DSP as usual and looked at them on the 1080p TV via the SD DVD player, comparing them to our usual best Compressor settings clip.

We found that your ‘Compressor basic’, ‘Compressor 2-pass’ and ‘out of AE MPEG2’ were all pretty much the same as our best Compressor settings clip, showing all the same artifacts when played on the 1080p TV. Adobe Media Encoder was close but a bit more noise.

Your BitVice was generally bad – very pixelly
‘HD Downscaled in AE to SD’ had more noise and colours looked washed out
Squeeze ‘DVD (PAL, PS)’ was an mpg and not accepted by DSP

You asked about the artifacts we’re seeing. On the 30s clip, made of 3 sequences, they are:-
Sequence 1
Jagged lines & noise - on man’s arm when he presses the buttons on the recorder, and on trouser legs, waistcoat
Ghosting – down the right hand side of man and on the music notes as they float up from the pipe

Sequence 2
Ghosting – to the right of most black lines eg the ratings poster in the BG (plus red line), man as he appears from side, desk
Noise - around man at desk’s face, especially glasses, the ratings poster in the background, telephone, the wording ‘Radio Clyde’ on t-shirt
Shimmer - on the arm & seat of the chair, paper tray, man from side
Jagged lines & noise – man from side’s trouser leg, waistcoat black

Sequence 3

Shimmer – on Big Ben, the windows of the buildings to the right of the scene
Jagged lines – on the wording of A-board esp ‘Jazz Club’, Jimmie’s guitar strings, guitar in window, posters on side
Noise – Jimmie’s face, clock, ‘new’ sin in window

All the same artifacts appeared on your Compressor & AE MPEG2s as on ours.

Again, when the DVD was played from the Mac hooked to the Dell UltraSharp 2408WFP these artifacts were not apparent.

Have you made a DVD and played the MPEGs at all yourself? If so, on what and what results did you observe?

Interested to hear what conclusions you draw from it all. For us it’s starting to look an awful lot like a problem between DVD SP and the end viewing - the TV set. If so, the question is why do other animation DVDs look okay but ours not? What else can we do?

Again, really really appreciate your time and help on this and look forward to hearing from you soon.

Kind regards,

Linda


Return to posts index

Jeff Greenberg
Re: Quality issues with animation using Compressor in creation of SD DVD
on Mar 22, 2011 at 3:11:47 am

Linda,

[Linda Chirrey] "Again, when the DVD was played from the Mac hooked to the Dell UltraSharp 2408WFP these artifacts were not apparent.
"


Now here's where it's interesting. Have you tried playing back the DVD in the mac via this connection? Instead of your dvd player?

[Linda Chirrey] "All the same artifacts appeared on your Compressor & AE MPEG2s as on ours.
"

I did mostly presets - I wanted to verify which if any encoders showed a difference. I didn't do much QC.


[Linda Chirrey] "Have you made a DVD and played the MPEGs at all yourself? If so, on what and what results did you observe?
"

Nope - I'm working 14 hour days already!


[Linda Chirrey] "Interested to hear what conclusions you draw from it all. For us it’s starting to look an awful lot like a problem between DVD SP and the end viewing - the TV set. If so, the question is why do other animation DVDs look okay but ours not? What else can we do?
"


I think so - but not DVDSP - I think it's your TV set.

I've been thinking about this all weekend..and besides the 'analog' nature of film, I wonder if any level of a motion blur would help.

But try the play out of the apple dvd player at the top of this post...

Best,

Jeff G

Apple Master Trainer | Avid Cert. Instructor DS/MC
Come See me speak at NAB!
Compressor Essentials from Lynda.com
(older but still good) Marquee, Media Composer (3.5) and Basic/Advanced Color DVDs (1.0) from Vasst.com
Contact me through my Website


Return to posts index

Linda Chirrey
Re: Quality issues with animation using Compressor in creation of SD DVD
on Mar 23, 2011 at 8:10:15 pm

Hi Jeff,

I hope you're well and have had some rest!!

[Jeff Greenberg] "Now here's where it's interesting. Have you tried playing back the DVD in the mac via this connection? Instead of your dvd player?"

I played the DVD from the Mac connected to the TV (via an HMI) and the results were indeed interesting. For all your MPEG2s that we got results equal to our best Compressor (Compressor basic & 2-pass, AE), plus our best Compressor, pretty much all the artifacts had disappeared and the image quality looked really very good. The ones that we had bad results for (BitVice, Adobe Media Encoder & AE downscaled) all still looked quite bad (albeit not AS bad).

So after all that, my feelings are that the MPEG2 transcoding is ok from Compressor and the DVD itself is also ok (so no issue with DSP). The problem is between the SD DVD player (a couple of years old consumer grade Pioneer) and the TV. A combination of the MPEG2 decoding by the DVD, subsequent interlacing, then deinterlacing and upscaling by the TV, maybe?

We shall have a look at it on a lower res 720 TV and an SD CRT to see what they do with it from the DVD player.

So after that, I guess it's all about trying to find out what filters or settings might reduce the artifacts created when HD res computer generated animation is played on an SD DVD/HD TV set up (which is still pretty common these days).

As you say, the animation we checked on the DVDs would have been mastered on film, so maybe that it part of it...

Let me know if you think any differently and have any further suggestions.

Take it easy and many thanks again for all your great help.

All the best,

Linda


Return to posts index


Jeff Greenberg
Re: Quality issues with animation using Compressor in creation of SD DVD
on Mar 23, 2011 at 8:49:56 pm

It feels like an episode of House, doesn't it?

Glad it worked out and that I could be of help.

Best,

Jeff G

Apple Master Trainer | Avid Cert. Instructor DS/MC
------------
You should follow me (filmgeek) on twitter. I promise to be nice.
Come See me speak at NAB!
Compressor Essentials from Lynda.com
(older but still good) Marquee, Media Composer (3.5) and Basic/Advanced Color DVDs (1.0) from Vasst.com
Contact me through my Website


Return to posts index

eoghan dunne
Re: Quality issues with animation using Compressor in creation of SD DVD
on Apr 26, 2012 at 10:13:39 pm

Hey guys, I'm having the same problems with my animation film when i put it onto DVD, its not a HD format, its a simple MPEG2, i'm getting the same white ghost lines around the animation lines and characters, also i'm getting the panning shimmers on the animation, the backgrounds dont get it at all though,
i exported the original sequences from Toonboom as .mov files, then used Premiere Pro to put them together, exported premiere pro film as .avi file, i am compressing using Sorenson Squeeze to MPEG2, the film runs fine on my computer but on the dvd its quality is lost,
Linda did you fix the problem in the end and could you help me? I'm in a crisis and i need to get the film out next week to festivals,
Would really appreciate it,
Thanks Eoghan.


Return to posts index

Linda Chirrey
Re: Quality issues with animation using Compressor in creation of SD DVD
on Apr 29, 2012 at 7:07:45 pm

Hi Eoghan,
The problem we had seemed to come from our DVD player which was old and not able to handle the playback of the encoding on the DVD. And also our TV was a brand new HD 1080, which made it look much worse.
Have you checked on another combination of DVD and TV?
If the DVD is fine on your computer it should be fine for anything else.
Let me know
Linda


Return to posts index


eoghan dunne
Re: Quality issues with animation using Compressor in creation of SD DVD
on Apr 29, 2012 at 9:51:32 pm

Ok thanks Linda, i have tried on 3 dvd players on the same tv,

maybe i need to try other combo's like you said,

i just wish it would play as simply as all the other DVDs we have play fine,

Probably most film festivals use the top high end equipment anyway so should be fine,
i have been trying to export in tiff sequences today and playing with other ideas too,
It seems to be the exact same problems as you film had,
thanks for replying,
All the best,
Eoghan :)

E Dunne


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]