x264 advanced settings - questions on how to reduce banding
I am new to this forum and also new to the art of encoding. I am having a severe problem with banding. I know I am not the first one and I have already read lots of posts about this and how to fight it. I would like your help to see if I can add anything to my workflow to improve the video quality or if I am doing something wrong here.
The source file is a 10bit avi v210 file with no visible banding (the sequence is 04min 44s long and has size of 46Gb) but I’ll show only the most problematic scene. Here’s a screenshot showing how it looks like in the 10bit source file (the picture has been stretched so it can exagerate the fading):
The first thing I made with this file was making it 8bit with dither loading a .avs file into Virtualdub using the following code:
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\ffms2.dll")
FFvideosource("D:\Arq\encoding\timelapse 30fps master 10bit.avi",enable10bithack=true)
Then, the output is a 8bit avi (52Gb size) with a zero (or very very little and way acceptable) banding visible as you can see in this screenshot:
So I used HandBrake 0.9.9.5530 - 64 bit version to encode a .mp4 file via x264. One important thing to note is: I am aiming the best quality possible that I can output for a 5Gb file. 5Gb is the largest size Vimeo will accept my video so that`s my limit. I don`t mind if the file takes 2 weeks to encode to gain just one pixel better. The video will be online for years to come so encoding time is definetly out of question. The important thing is the quality above everything that can fit this size.
Picture - Anamorphic strict (1920x1080), custom croppping (0 for top, left, right and bottom).
Filters - Detelecine default, Ddecomb default. Deinterlace, Denoise and Deblock Off.
Video - Codec x264. Framerate constant (30fps in the source and the output). 2-pass encoding with average bitrate 145000 (that will give me a file around 4.9Gb).
Advanced - ref=16:bframes=16:b-adapt=2:direct=auto:me=tesa:subme=11:merange=64:an alyse=all:trellis=2:deblock=-1,-1
Here`s a screenshot of the advanced tab on Handbrake:
This settings takes around 12h to encode. Fine for me.
This is the output as seen from virtualdub:
And this is the same file but played with Media Classic Player:
When I play the 8bit dithered uncompressed file on Media Player Classic, the output range is 0-255. x264 changed the output it to 16-235.
Question #1: Is this normal?
I had to set the output range on MPC to 16-235 to match virtualdub colors. But on virtualdub absolute black is darker.
Question #2: Is Virtualdub transforming my video file in 0-255 to preview it?
Question #3: Does Vimeo do the same, i.e. expect a 16-235 and transform into 0-235? I had problems with gamma shift before, when uploading to vimeo, so it`s an important thing to consider. Is x264 outputting a 16-235 file the correct workflow for vimeo? (PS. I have read vimeo guidelines, and there`s no metion for the output range)
The MPC screenshot above is just an extra: all of the screenshots below will be made using Virtualdub so we can compare with the 10bit file.
As you can see, obvious and strong banding after x264, even with dither. The problem seem to be on x264. So I started reading on which values to tweak to try to improve this. I have read a lot and have made a few experiments.
Firstly, I changed AQ Streght to 0.8 and add the no-mbtree parameter. Here`s a sreenshot on the output file with these changes:
Not a change. And then I also set deblock to 3,3, used the No-DCT-Decimate with the AQ strenght 0,8 and no-mbtree. The result is:
Not a change also.
Question #4: What I am doing wrong here? Anyhow I can tweak x264 to make this fading better with less banding?
I don`t know what else I can do to reduce this. I don`t think adding noise to the source file will help, since the 8bit source has already no banding.