HDTV (Rec. 709) vs Rec.709 Gamma 2.4 - Video Output
I'm experiencing a very confusing issue with color working space and output. I don't know if it's intended, a bug, or I'm misunderstanding something.
The best part is... if I set the OUTPUT color profile to Rec.709 gamma 2.4 it perfectly matches the project working space of HDTV (Rec.709)
But If I set the color space of the Project to Rec.709 2.4 and not HDTV (Rec.709) the test chart looks off.
I know ProRes has a history of horrible gamma issues with interpretation in applications, but the fact that I need to set a different output color profile to match the project working color space has me spooked.
Am I misunderstanding something? DNxHD has the same problem from what I can tell. We need to render to a video format so postings can be sent without re-compiling an image sequence (that doesn't have this problem for some reason).
Any insight is greatly appreciated.
The problem with outputting using a 2.4 gamma "709" profile is that Rec709 is not, and has never been, a source-referred signal of gamma 2.4. The 2.4 gamma is a display profile not an encoding profile.
The SOURCE REFERRED (i.e. camera) gamma is a parametric curve that is roughly the inverse gamma of 1.9-2.0 (i.e 1/2.0 for a curve of 0.5).
This is intended to play through a display that has a gamma higher than 2.0, although the actual display specs are NOT part of the Rec709 specification. When Rec709 was originally developed (circa 1990) playback was through CRT displays which tend to have an output gamma of 2.2-2.4 or so, depending on how the contrast and brightness is set.
This results in a total system gamma of around 1.2, and NOT a purely linear 1.0 in-to-out transfer. This is done to preserve the perception of the scene's contrast while viewing in a darkened living room. It wasn't until a few years ago that the ITU finally defined a standard for viewing (rec1886) that defines a display monitor with a gamma of 2.4, and Rec2035 which defines the viewing conditions of said darkened living room. (and just a few years before that, EBU tech3325 stated it should be gamma 2.28).
But the Rec709 SIGNAL spec (i.e. the way it is encoded in the output file) has NOT CHANGED - it is still a parametric curve with a gamma close to 1/2.0 — this means it is wrong to encode it with a 1/2.4 (0.42) gamma. It should still be encoded with a ~0.5 curve.
PROFILE ISSUES and 16-235 LEVELS:
There are two other things that may be tripping you up here. First of all, there are a lot of Rec709 profile floating around out there, and they are not consistent, some are "simplified", and some are just wrong. This includes profiles that ship with Adobe products.
But also: black and white levels are NOT 0 and 255. The Rec709 spec defines that Black is 16 and white is 235 for 8 bit, and 64/940 for 10 bit. NTSC (Rec601) was the same way. But After Effects usually works with black at 0 and white at 255 (or 0.0 and 1.0 in 32 bit), this is known as "full range". On output, black is supposed to be mapped to 16 for 8 bit or 64 for 10 bit "video" files. While in the spec, black is allowed to go lower than 16 (super black) and white higher than 235 (superwhite). But as far as the specification is concerned, 16 should be the darkest color on the monitor, and 235 should be the brightest.
You'll notice in AE when you bring in ProRes 422 footage, you can't even change the color interpretation:
And it's technically not an "embedded" profile — it's a "calibrated" profile, that is, it's a "standard" set by the ITU, and AE is "interpreting" it. And despite having the "interpretationRules.txt" file, YUV / Y'CbCr type files don't allow you to choose the INPUT colorspace in AE.
Legal Video Levels:Even the Arri Alexa when shooting in LogC mode aims for the legal levels. The reason the numbers don't indicate a black at 16 inside After Effects is that AE maps 16 to 0 if you setup your workspace inside AE as a 0-255 space.
So in essence AE "hides" the true levels being input/output from the system. And unfortunately their use of terminology and the way the dialogs are set up makes it all the more confusing.
COLOR MANAGED vs COLOR CALIBRATED WORKFLOWS
What is important is to MONITOR your workspace with a 2.4 display emulation, I'll discuss this later under "workflow", but it is important to remember that normally playback is not "color managed" it is "color calibrated", which is quite different in meaning.
Remember that color TV and film were around a long time before ICC color management. Those workflows managed (and still do today) because the camera, display, and the broadcast chain in between are all calibrated to a set standard of color and transfer curves, etc.
ICC color management is very different — the idea is to convert the digital color values of image data from one color space, then into an intermediate "connection" colorspace (usually CIEXYZ) that is neutral, and then converting it to the other space. There's a ton of math to make that happen which is why AE slows down so much when color management is on), it's more efficient to just have everything calibrated, as it is in a broadcast chain.
One advantage of color management is the ability to emulate different forms of output using Display Color Management, where your calibrated and properly profiled monitor can display different view conditions, such as Rec1886, while working in a completely different space like 32 bit Linear Gamma (1.0) and P3 primaries.
Side note: if you are using Display Color Management, checking/unchecking "Compensate for Scene Referred Profiles" will cause a Rec709 clip to appear visibly lighter/darker — but it does not change the RGB values.
Which brings us back to profiles. The 2.4 gamma Rec709 profile is a VIEWING profile, useable for proofing or emulating an output monitor. It is for display only, it is not a profile to be used for encoding the TRC into your output (unless you specifically want to "bake in" that crunched look).
An example workflow is (using a Linear gamma 1.0 workflow with a Rec709 output target).
1) Set the project colorspace to 32 bit Linear, and an unbounded sRGB or Rec709 profile (sRGB and Rec709 have identical primaries and whitepoint). Turn ON "Compensate for Scene Referred Profiles".
I recommend Elle's "Well Behaved" V4 profiles, downloadable here: https://github.com/ellelstone/elles_icc_profiles/tree/master/profiles I've found these to be the most accurate of profiles, with true Rec709 TRCs.Setting for workspace:
2) Setup a custom Display Simulation, using Rec709-elle-V4-rec709.iccas the first output stage, set "Preserve RGB" so that the display profile (a Rec1886 aka a Rec709 @2.4 gamma) takes this data and then displays it the way a 2.4 gamma monitor would.
Setting for simulation:
3) Now do all your work — when you output, you output to the Rec709-elle-V4-rec709.icc profile, which put on the standard Rec709 TRC, but it will look the way you were working on it when you send it through a 2.4 gamma reference monitor. Essentially, once you output the video, color management is over. Downstream it is being played back in non-managed (but hopefully calibrated) displays.
This ended up a bit longer than I intended, but I hope it was helpful. This very question/problem comes up about every two weeks, partly I think because Adobe isn't doing a very good job explaining how to use their color management. Adobe still has tutorials online suggesting using ProPhoto as a working space for film and video, which is a terrible space to use — it uses a D50 Illuminant, while Most film and video is based (or very close) on D65, and ProPhoto is way too big and creates negative color values and imaginary colors that can wrap around and result in bad values in DPX outputs and other unpredictable garbage.
VFX & Title Supervisor
Hi Andrew Somers,
Thank you for the detailed info. May I know what is your recommended workflow for DCI P3 output for AE?
The short answer is "it depends" as I don't know anything about your project — what are your source files, what are you rendering to, and where are those files going next, and what does the DI need, etc etc. I can't answer your question without specifics.
Here is an article I wrote some time ago on workflow considerations that might be helpful:
Scene Referred Colorspace
In general, today I mostly recommend not using ICC color management for video/film projects. Instead OpenColorIO is a more useful tool for the job, and it's free.
VFX & Title Supervisor