HEVC test content validation results summary

[nicholas-fr]

Following updates to the validation scripts, here are results for the HEVC test content as defined here:
https://docs.google.com/spreadsheets/d/1Bmgv6-cfbWfgwn7l-z0McUUI1rMjaWEwrN_Q30jaWk4/

Attached are easier to read spreadsheet overviews, as well as the full JSON results:
hevc_12.5_25_50_test_results_2025-02-27_12-10-13.xlsx
hevc_14.985_29.97_59.94_test_results_2025-02-27_12-10-13.xlsx
hevc_15_30_60_test_results_2025-02-27_12-10-13.xlsx
test-content-validation-hevc_20250227-json.zip

Not yet validated: switching sets, encrypted streams, chunked content. I aim to address that next week.

In addition to these issues, there are also JCCP DASH validator failures to address (included in the JSON results) for all test streams.

15_30_60 test content failures:

## Stream 15_30_60_t2_2025-01-15:  chh1 2
Test: codec_tier                              FAIL      High instead of Main
Test: b_frames_present                        FAIL      False instead of True
Test: duration                                FAIL      29.75 instead of 30


## Stream 15_30_60_t3_2025-01-15:  chh1 3
Test: codec_tier                              FAIL      High instead of Main


## Stream 15_30_60_t6_2025-01-15:  chh1 6
Test: b_frames_present                        FAIL      False instead of True


## Stream 15_30_60_t11_2025-01-15:  chh1 11
Test: cmf2_sample_flags_present               FAIL      False instead of True


## Stream 15_30_60_splice_ad_2025-01-15:  chd1 splice_ad
Test: cmaf_fragment_duration                  FAIL      1.6 instead of 2.0
Test: duration                                FAIL      6.4 instead of 5.76


## Stream 15_30_60_t22_2025-01-15:  cud1 22
Test: codec_tier                              FAIL      High instead of Main
Test: b_frames_present                        FAIL      False instead of True
Test: duration                                FAIL      29.75 instead of 30


## Stream 15_30_60_t31_2025-01-15:  cud1 31
Test: cmf2_sample_flags_present               FAIL      False instead of True


## Stream 15_30_60_t32_2025-01-15:  cud1 32
Test: cmf2_sample_flags_present               FAIL      False instead of True


## Stream 15_30_60_t41_2025-01-15:  clg1 41
Test: cmf2_sample_flags_present               FAIL      False instead of True


## Stream 15_30_60_t42_2025-01-15:  clg1 42
Test: codec_tier                              FAIL      High instead of Main
Test: vui_transfer_characteristics            FAIL      14 instead of 18
Test: duration                                FAIL      29.75 instead of 30
Test: mpd_bitstream_mismatch                  FAIL      AdaptationSet.SupplementalProperty -> TransferCharacteristics=18; instead of AdaptationSet.SupplementalProperty -> TransferCharacteristics=;
NOTE: preferred transfer characteristics SEI are not present so this is not signalled as HLG


## Stream 15_30_60_t50_2025-01-15:  clg1 50
Test: cmf2_sample_flags_present               FAIL      False instead of True


## Stream 15_30_60_t62_2025-01-15:  chd1 62
Test: codec_tier                              FAIL      High instead of Main
Test: b_frames_present                        FAIL      False instead of True
Test: duration                                FAIL      29.75 instead of 30


## Stream 15_30_60_t71_2025-01-15:  chd1 71
Test: cmf2_sample_flags_present               FAIL      False instead of True


## Stream 15_30_60_t72_2025-01-15:  chd1 72
Test: cmf2_sample_flags_present               FAIL      False instead of True


## Stream 15_30_60_t73_2025-01-15:  chd1 73
Test: cmf2_sample_flags_present               FAIL      False instead of True

14.985_29.97_59.94 test content failures:

## Stream 14.985_29.97_59.94_t2_2025-01-15:  chh1 2
Test: codec_tier                              FAIL      High instead of Main
Test: b_frames_present                        FAIL      False instead of True
Test: duration                                FAIL      29.746 instead of 30


## Stream 14.985_29.97_59.94_t3_2025-01-15:  chh1 3
Test: codec_tier                              FAIL      High instead of Main


## Stream 14.985_29.97_59.94_t6_2025-01-15:  chh1 6
Test: b_frames_present                        FAIL      False instead of True


## Stream 14.985_29.97_59.94_t11_2025-01-15:  chh1 11
Test: cmf2_sample_flags_present               FAIL      False instead of True


## Stream 14.985_29.97_59.94_splice_ad_2025-01-15:  chd1 splice_ad
Test: cmaf_fragment_duration                  FAIL      1.93 instead of 2.0
Test: cmf2_sample_flags_present               FAIL      False instead of True
Test: duration                                FAIL      21.255 instead of 5.76


## Stream 14.985_29.97_59.94_t22_2025-01-15:  cud1 22
Test: codec_tier                              FAIL      High instead of Main
Test: b_frames_present                        FAIL      False instead of True
Test: duration                                FAIL      29.746 instead of 30


## Stream 14.985_29.97_59.94_t31_2025-01-15:  cud1 31
Test: cmf2_sample_flags_present               FAIL      False instead of True


## Stream 14.985_29.97_59.94_t32_2025-01-15:  cud1 32
Test: cmf2_sample_flags_present               FAIL      False instead of True


## Stream 14.985_29.97_59.94_t42_2025-01-15:  clg1 42
Test: codec_tier                              FAIL      High instead of Main
Test: vui_transfer_characteristics            FAIL      14 instead of 18
Test: b_frames_present                        FAIL      False instead of True
Test: duration                                FAIL      29.746 instead of 30
Test: mpd_bitstream_mismatch                  FAIL      AdaptationSet.SupplementalProperty -> TransferCharacteristics=18; instead of AdaptationSet.SupplementalProperty -> TransferCharacteristics=;
NOTE: preferred transfer characteristics SEI are not present so this is not signalled as HLG


## Stream 14.985_29.97_59.94_t62_2025-01-15:  chd1 62
Test: codec_tier                              FAIL      High instead of Main
Test: b_frames_present                        FAIL      False instead of True
Test: duration                                FAIL      29.746 instead of 30


## Stream 14.985_29.97_59.94_t71_2025-01-15:  chd1 71
Test: cmf2_sample_flags_present               FAIL      False instead of True


## Stream 14.985_29.97_59.94_t72_2025-01-15:  chd1 72
Test: cmf2_sample_flags_present               FAIL      False instead of True


## Stream 14.985_29.97_59.94_t73_2025-01-15:  chd1 73
Test: cmf2_sample_flags_present               FAIL      False instead of True

12.5_25_50 test content failures:

## Stream 12.5_25_50_t2_2025-01-15:  chh1 2
Test: codec_tier                              FAIL      High instead of Main
Test: b_frames_present                        FAIL      False instead of True
Test: duration                                FAIL      29.7 instead of 30


## Stream 12.5_25_50_t3_2025-01-15:  chh1 3
Test: codec_tier                              FAIL      High instead of Main


## Stream 12.5_25_50_t6_2025-01-15:  chh1 6
Test: b_frames_present                        FAIL      False instead of True


## Stream 12.5_25_50_t10_2025-01-15:  chh1 10
Test: cmf2_sample_flags_present               FAIL      False instead of True


## Stream 12.5_25_50_t11_2025-01-15:  chh1 11
Test: cmf2_sample_flags_present               FAIL      False instead of True


## Stream 12.5_25_50_splice_ad_2025-01-15:  chd1 splice_ad
Test: cmaf_fragment_duration                  FAIL      1.92 instead of 2.0


## Stream 12.5_25_50_t22_2025-01-15:  cud1 22
Test: codec_tier                              FAIL      High instead of Main
Test: b_frames_present                        FAIL      False instead of True
Test: duration                                FAIL      29.7 instead of 30


## Stream 12.5_25_50_t31_2025-01-15:  cud1 31
Test: cmf2_sample_flags_present               FAIL      False instead of True


## Stream 12.5_25_50_t32_2025-01-15:  cud1 32
Test: cmf2_sample_flags_present               FAIL      False instead of True


## Stream 12.5_25_50_t42_2025-01-15:  clg1 42
Test: codec_tier                              FAIL      High instead of Main
Test: vui_transfer_characteristics            FAIL      14 instead of 18
Test: b_frames_present                        FAIL      False instead of True
Test: duration                                FAIL      29.7 instead of 30
Test: mpd_bitstream_mismatch                  FAIL      AdaptationSet.SupplementalProperty -> TransferCharacteristics=18; instead of AdaptationSet.SupplementalProperty -> TransferCharacteristics=;
NOTE: preferred transfer characteristics SEI are not present so this is not signalled as HLG


## Stream 12.5_25_50_t62_2025-01-15:  chd1 62
Test: codec_tier                              FAIL      High instead of Main
Test: b_frames_present                        FAIL      False instead of True
Test: duration                                FAIL      29.7 instead of 30


## Stream 12.5_25_50_t71_2025-01-15:  chd1 71
Test: cmf2_sample_flags_present               FAIL      False instead of True


## Stream 12.5_25_50_t72_2025-01-15:  chd1 72
Test: cmf2_sample_flags_present               FAIL      False instead of True


## Stream 12.5_25_50_t73_2025-01-15:  chd1 73
Test: cmf2_sample_flags_present               FAIL      False instead of True

[nlsdvl]

I believe all issues have been adressed. Looking forward to third party validation. cta-wave/Test-Content#66
the following have been updated: t2, t3, t6, t22, t31, t32, t42, t62

• update individual test vectors once issues are fixed
• update related chunked content
• spliced content
• update related switching sets (cud1)

[rbouqueau]

@nicholas-fr Could you please check all is ok for you if you haven't already?

[rbouqueau]

@nicholas-fr Could you please check all is ok for you if you haven't already?

[rbouqueau]

@nicholas-fr Could you please check all is ok for you if you haven't already?