Webcodecs: Handle limited range pixel values in NV12 data.

[scriby]

NV12 data coming from the browser's VideoDecoder API may be using a different range of pixel values (instead of 0-255). The pixel ranges need to be re-mapped to use the full range expected by libheif.

In particular, this caused transparent backgrounds to be light gray due to the bottom end of the range starting at 16 instead of 0.

This PR also adds explicit handling of the mono alpha channel that's used to encode the transparency information in HEIC images when dealing with NV12 data.

P.S.: Not related to the changes in this PR, but I noticed that the alpha channel does not work properly when decode_with_browser_hevc returns RGBA data. I tried a few different ways to get it to work, but wasn't able to (the background is always black). I'm not sure if libheif is setup to handle the alpha channel mask when using RGBA data, or if there was an issue with my approach.

I was able to get it to work by converting RGBA data to NV12 and reusing that pathway. I know you didn't like that approach before, but it has a bit more meaning now to reuse the code b/c there's more custom code on each pathway for handling the mono/transparency mask. Let me know if you want a PR for that or if you want to try to work on getting alpha support to work with RGBA formatted data.

Note that from my perspective it is not that big of a problem, because I haven't yet seen any images with transparency that are using the code path that returns RGBA (I have to "force" it by modifying the code). But it might possibly be more likely on other hardware that I'm not testing on.

For my purposes, the webcodecs plugin is already very useful as it is, b/c HEIC with alpha channels are already somewhat rare to begin with.

[silverbacknet]

There are two different modes in video that rarely existed in images, but now that video and image share formats, they have become relevant. What's often called full-range, PC-mode, or JPEG-mode is from 0-255, while limited-range, TV-mode, or video-mode is from 16-235 (16-240 chroma), with those numbers scaled up for 10+ bit. The latter is by far the most common for any video you'll encounter, the former for images.

Libheif supports both via the full_range_flag in nclx, and will flag an image as being in limited or full range -- the decoder should handle that. (Which decoders have been getting much better about actually doing.) I think you can override it to get the right color conversion even when decoding. That saves the hit of an extra color conversion step plus additional round-off errors from the scaling.

[scriby]

I tried to use that information to update the approach, but wasn't able to get something fully working.

Here's what I tried:

Create the nclx color profile:

  struct heif_color_profile_nclx* nclx = heif_nclx_color_profile_alloc();
  nclx->color_primaries = get_heif_primaries(primaries);
  nclx->transfer_characteristics = get_heif_transfer(transfer);
  nclx->matrix_coefficients = get_heif_matrix(matrix);
  nclx->full_range_flag = is_full_range ? 1 : 0;
  heif_image_set_nclx_color_profile(*out_img, nclx);
  heif_nclx_color_profile_free(nclx);

Implemented conversion functions to convert from the values returned by the browser's decoded VideoFrame:

static heif_color_primaries get_heif_primaries(const std::string& p) {
  if (p == "bt709") return heif_color_primaries_ITU_R_BT_709_5;
  if (p == "smpte170m") return heif_color_primaries_ITU_R_BT_601_6;
  // output truncated for simplicity
}

static heif_transfer_characteristics get_heif_transfer(const std::string& t) {
  if (t == "bt709") return heif_transfer_characteristic_ITU_R_BT_709_5;
  if (t == "smpte170m") return heif_transfer_characteristic_ITU_R_BT_601_6;
  // output truncated for simplicity
}

static heif_matrix_coefficients get_heif_matrix(const std::string& m) {
  if (m == "bt709") return heif_matrix_coefficients_ITU_R_BT_709_5;
  if (m == "smpte170m") return heif_matrix_coefficients_ITU_R_BT_601_6;
  // output truncated for simplicity
}

It would also be possible to grab this data from the VUI section of the SPS data, but the SPS parsing function we've got in this file doesn't have code to parse the VUI info.

I printed some debug output to make sure this was all getting set properly, and I saw primaries = smpte170m (6), transfer = bt709 (1), matrix = smpte170m (6), and full range = 0.

All these values seem correct to me, so I'm not quite sure why it doesn't work. Is there anything special you have to do when using the image data, such as over in post.js?

[scriby]

I've investigated this further and discovered two things that prevent setting of the nclx profile from working as expected.

• The color conversion functions in yuv2rgb.cc don't re-map the alpha channel from limited to full range. This causes fully transparent alpha channels to be light gray even when a conversion function that handles limited range data is selected.

• I am seeing that even if I return nclx data on the decoded image, libheif may ignore it. I tried to dig into this and I think what's going on is that if a HEIF image contains nclx data in the colr box, it will be used instead of the nclx data from the decoded tiles. This can cause it to incorrectly think that the image is using full range, even if the decoded image data is using limited range. I am testing with an image generated from an iPhone that has an alpha channel and am seeing that Op_YCbCr420_to_RGB32 is being selected for color conversion even though the decoded tiles are in limited range and Op_YCbCr420_to_RGB32 only handles full range. In particular I am seeing that decoder_webcodecs.cc is setting primaries = 6, transfer = 1, matrix = 6, full range = 0 but yuv2rgb.cc is instead seeing primaries = 1, transfer = 13, matrix = 6, full range = 1.

After this investigation I can see that it doesn't make sense to do the limited range color conversion in the decoder plugin. So I will go ahead and update this PR just to properly set the nclx profile info on the decoded image.

However, I still think both things I raised above should be addressed to make color conversion & preserve the alpha channel properly for data returned from the webcodecs plugin. If there's any direction on that and whether you'd like me to take either thing on let me know.

Thanks!