Skip to content

Fix H.265 decoder crushing 10-bit output and discarding bitstream colorimetry (HDR10 renders washed out) #267

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
oliversluke wants to merge 3 commits into
base: m144_release
Choose a base branch
from
+86 −7
Open

Fix H.265 decoder crushing 10-bit output and discarding bitstream colorimetry (HDR10 renders washed out) #267

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e79d430
Fix H.265 decoder crushing 10-bit output and discarding bitstream col…
oliversluke Jul 9, 2026
d4af43d
Capture bitstream colour info per frame instead of reading decoder st…
oliversluke Jul 10, 2026
ff6af96
Gate 10-bit decode output behind an opt-in class property
oliversluke Jul 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 44 additions & 3 deletions sdk/objc/components/video_codec/RTCVideoDecoderH265.mm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
@interface RTC_OBJC_TYPE (RTCVideoDecoderH265) ()
- (void)setError:(OSStatus)error;
- (void)processFrame:(RTC_OBJC_TYPE(RTCVideoFrame) *)decodedFrame reorderSize:(uint64_t)reorderSize;
- (bool)bitstreamCarriesColorInfo;
@end

static void overrideColorSpaceAttachments(CVImageBufferRef imageBuffer) {
Expand Down Expand Up @@ -183,7 +184,14 @@ void h265DecompressionOutputCallback(void *decoderRef, void *params, OSStatus st
return;
}

overrideColorSpaceAttachments(imageBuffer);
// Only guess colour attachments when the bitstream signalled none. Streams
// that carry VUI colour information (e.g. HDR10: PQ transfer + BT.2020
// primaries) already have correct attachments propagated by VideoToolbox
// from the format description; overriding them mistags the frames as
// BT.709/sRGB and HDR content renders washed out.
if (![decoder bitstreamCarriesColorInfo]) {

@pblazej pblazej Jul 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This runs on VideoToolbox's async callback thread and reads _videoFormat, which decodeData: releases (via setVideoFormat:) before destroyDecompressionSession waits for in-flight frames — a use-after-free window on every mid-stream format change; consider capturing this as a bool in RTCH265FrameDecodeParams at decode time instead.

@oliversluke oliversluke Jul 10, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — the callback could indeed read _videoFormat in the window where decodeData: had already released the old format while the previous session's frames were still in flight (and even after the swap, in-flight frames would be judged against the wrong format's colour info). Fixed as you suggested in d4af43d: the flag is now captured in RTCH265FrameDecodeParams on the decode thread, so the callback no longer touches decoder state and each frame is judged against the format it was actually decoded with.

overrideColorSpaceAttachments(imageBuffer);
}

// TODO(tkchin): Handle CVO properly.
RTC_OBJC_TYPE(RTCCVPixelBuffer) *frameBuffer =
Expand Down Expand Up @@ -420,8 +428,13 @@ - (int)resetDecompressionSession {
#endif
kCVPixelBufferIOSurfacePropertiesKey, kCVPixelBufferPixelFormatTypeKey};
CFDictionaryRef ioSurfaceValue = CreateCFTypeDictionary(nullptr, nullptr, 0);
int64_t nv12type = kCVPixelFormatType_420YpCbCr8BiPlanarFullRange;
CFNumberRef pixelFormat = CFNumberCreate(nullptr, kCFNumberLongType, &nv12type);
// Forcing NV12 would crush high bit depth output (e.g. HEVC Main10) to
// 8 bits. Request a 10-bit biplanar format for such streams; 8-bit streams
// keep NV12 so existing consumers are unaffected.
int64_t pixelFormatType = [self isHighBitDepthFormat]
? kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange
: kCVPixelFormatType_420YpCbCr8BiPlanarFullRange;
CFNumberRef pixelFormat = CFNumberCreate(nullptr, kCFNumberLongType, &pixelFormatType);
CFTypeRef values[attributesSize] = {kCFBooleanTrue, ioSurfaceValue, pixelFormat};
CFDictionaryRef attributes = CreateCFTypeDictionary(keys, values, attributesSize);
if (ioSurfaceValue) {
Expand Down Expand Up @@ -453,6 +466,34 @@ - (void)configureDecompressionSession {
VTSessionSetProperty(_decompressionSession, kVTDecompressionPropertyKey_RealTime, kCFBooleanTrue);
}

// Returns true when the active format description signals a luma bit depth
// above 8 (e.g. HEVC Main10), read from bit_depth_luma_minus8 in the hvcC
// decoder configuration record.
- (bool)isHighBitDepthFormat {
if (!_videoFormat) {
return false;
}
CFDictionaryRef atoms = (CFDictionaryRef)CMFormatDescriptionGetExtension(
_videoFormat, kCMFormatDescriptionExtension_SampleDescriptionExtensionAtoms);
if (!atoms) {
return false;
}
CFDataRef hvcc = (CFDataRef)CFDictionaryGetValue(atoms, (CFStringRef) @"hvcC");
if (!hvcc || CFDataGetLength(hvcc) < 18) {
return false;
}
return (CFDataGetBytePtr(hvcc)[17] & 0x07) > 0;
}

// Whether the active format description carries colour information parsed
// from the bitstream (VUI colour description). VideoToolbox propagates these
// extensions onto output pixel buffers, so no fallback tagging is needed.
- (bool)bitstreamCarriesColorInfo {
return _videoFormat &&
CMFormatDescriptionGetExtension(_videoFormat, kCMFormatDescriptionExtension_ColorPrimaries) !=
nullptr;
}

- (void)destroyDecompressionSession {
if (_decompressionSession) {
VTDecompressionSessionWaitForAsynchronousFrames(_decompressionSession);
Expand Down