ISO 21496-1 / Ultra HDR gain map rendering support

[Zemax27]

Expected behavior of the wanted feature

mpv currently renders Ultra HDR / ISO 21496-1 JPEG images as plain SDR, silently discarding the embedded gain map. This request tracks the full pipeline needed for correct adaptive HDR rendering of these images, which requires coordinated work at the FFmpeg and libplacebo layers.

What is a gain map image

ISO 21496-1:2025 (jointly adopted by Apple as "Adaptive HDR" and Google as "Ultra HDR") defines a container format where a JPEG file embeds:

• a base SDR image (standard JPEG, fully backward-compatible)
• a gain map image (second JPEG, located via the MPF/APP2 directory)
• gain map metadata: min_content_boost, max_content_boost, gamma, channel offsets

The gain map encodes the difference between the SDR base and the HDR rendition. At render time, the blending weight w is derived from the actual target display headroom:

w = clamp(
      (log2(display_headroom) - log2(min_content_boost))
    / (log2(max_content_boost) - log2(min_content_boost)),
    0.0, 1.0
)

log_gain = mix(gainmap_min, gainmap_max, gainmap_value ^ (1/gamma)) * w
hdr_linear = sdr_linear * exp2(log_gain)

The key property is that display_headroom is resolved at render time against the actual display. This is fundamentally different from static HDR10 metadata and cannot be correctly handled by pre-baking a headroom value at decode time.

Current behavior

When opening an Ultra HDR JPEG in mpv (any version, any --vo), only the SDR base image is decoded and displayed. The MPF-embedded gain map is never read. On an HDR display with --vo=gpu-next, the image is displayed at SDR brightness with no tone expansion.

This is correct behavior given the current state of the stack, but it means a large and growing fraction of real-world images (every photo taken on Android >=15 or iOS >=18) is displayed incorrectly on HDR hardware.

Required changes by layer ( possible implementation )

This is a multi-layer problem. Neither FFmpeg nor libplacebo can solve it alone.

Layer 1: FFmpeg libavcodec/mjpegdec (extraction)

FFmpeg must be extended to:

• Parse the MPF (Multi-Picture Format) APP2 directory in JPEG files to locate the embedded gain map image
• Decode the gain map as a secondary JPEG into a pixel buffer
• Parse the ISO 21496-1 APP2 binary metadata block (or legacy XMP fallback) into AVGainMapMetadata (the struct already exists in libavutil)
• Attach gain map pixels and metadata to the primary decoded frame as AV_FRAME_DATA_GAIN_MAP + AV_FRAME_DATA_GAIN_MAP_METADATA side data

This is purely extraction and passthrough. No blending, no HDR conversion. The SDR base frame flows downstream unchanged; the gain map rides along as side data.

Layer 2: libplacebo (application)

libplacebo, which already has full access to runtime display HDR capabilities via pl_hdr_metadata (populated from EDID), is the correct place to apply the gain map formula. It needs:

• A new pl_gain_map side data type to carry gain map pixels (pl_tex) and metadata
• Detection of gain map side data on an incoming pl_frame
• In the render pass: bilinear upsampling of the gain map texture (it is often half-resolution), application of the ISO 21496-1 blend formula in linear light using the live display_headroom from pl_hdr_metadata.max_luma, integrated before tone mapping

Applying the blend inside the existing libplacebo shader pass is architecturally correct and avoids any extra transcoding round-trip. The result feeds naturally into the existing HLG/PQ/tone mapping pipeline.

Layer 3: mpv (wiring)

Once the above two layers are in place, mpv needs to:

• Forward AV_FRAME_DATA_GAIN_MAP + AV_FRAME_DATA_GAIN_MAP_METADATA side data from the FFmpeg-decoded AVFrame through to the pl_frame passed to libplacebo
• This follows the same pattern already used for AV_FRAME_DATA_MASTERING_DISPLAY_METADATA and AV_FRAME_DATA_CONTENT_LIGHT_LEVEL

The mpv change is expected to be small; the heavy lifting is in FFmpeg and libplacebo.

NOTE for libplacebo

The gain-map has a format's core adaptive property: the same image should render dimmer on an SDR display, brighter on a 1000-nit display, and even brighter on a 4000-nit display, all from the same file, by resolving w against the actual display at render time. Only libplacebo has that information and it should be also compatible with mpv options --target-peak and --hdr-reference-white.

Scope

• Primary target: JPEG-based gain maps (Ultra HDR / ISO 21496-1 over JPEG), as produced by Android >= 15, iOS >= 18, and recent camera firmware (e.g. Sigma BF)
• If possible: HEIF/AVIF-based gain maps (Apple HEIC), though the libplacebo side would be shared
• --vo=gpu-next only; software vo paths are out of scope

Reference implementations

• google/libultrahdr — C++ reference, applyGainMap / applyGainCore
• imazen/ultrahdr — Rust port, ISO 21496-1 APP2 parser + gain map application, parity-tested against libultrahdr
• libavif src/gainmap.c — precedent for gain map handling in a libav-adjacent project
• Chromium components/viz/common/gpu/gain_map_decoder.cc — production browser implementation

Alternative behavior of the wanted feature

No response

Log File

No response

Sample Files

No response