Possible RTP Timestamp Corruption in Kurento 7.2.0 Causing Severely Incorrect HLS Metadata

[y-sugitani]

Prerequisites

These are MANDATORY, otherwise the issue will be automatically closed.

• I agree to fill this issue template.
• I have read the Troubleshooting Guide and Support Instructions.

Issue description

There appears to be a serious issue with RTP transmission in Kurento 7.2.0. When sending WebRTC input to FFmpeg via RTP endpoints for HLS output, the resulting TS files have severely corrupted metadata. In particular, the timestamps and duration information are completely wrong.

Specifically:

• The metadata duration is vastly different from the actual content length (metadata shows several hours for content that is only a few seconds long)
• Audio and video timestamps are significantly out of sync (approximately 22 hours difference)
• Timestamp continuity between consecutive segments is not maintained

This issue did not occur in Kurento 6 and only appeared after migrating to Kurento 7, suggesting that the change in timestamp units from seconds to milliseconds might be the root cause.

Context

We are developing a service that composites WebRTC calls and sends them to streaming servers (like Wowza) via RTP for HLS distribution. Everything worked correctly with Kurento 6, but after upgrading to Kurento 7, these issues have emerged.

This severely impacts the quality of our generated HLS streams and affects our service quality.

How to reproduce?

1. Prepare the following SDP and save it to a file (test.sdp):

v=0
o=- 0 0 IN IP4 192.168.1.217
s=Streaming Session
c=IN IP4 192.168.1.217
t=0 0
m=video 5004 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 profile-level-id=42e01f;packetization-mode=1
a=recvonly
m=audio 5006 RTP/AVP 97
a=rtpmap:97 opus/48000/2
a=recvonly

2. Run the following FFmpeg command to listen for RTP connections:

ffmpeg -protocol_whitelist file,udp,rtp -i test.sdp \
    -map 0:v -c:v copy \
    -map 0:a -c:a aac -b:a 128k \
    -f hls \
    -hls_time 4 \
    -hls_playlist_type event \
    -hls_segment_filename "hls/stream_%03d.ts" \
    hls/playlist.m3u8

3. Build one of the following pipelines in Kurento:
   a. WebRtcEndpoint -> RtpEndpoint (sending to IP and ports specified in the SDP)
   b. WebRtcEndpoint -> Composite -> RtpEndpoint (sending to IP and ports specified in the SDP)

4. Connect a WebRTC client to Kurento and start streaming

5. Analyze the generated HLS TS files with ffprobe to observe timestamp and duration anomalies

Expected & current behavior

Expected behavior:

• Each TS segment's metadata matches the actual content length
• Audio and video timestamps are synchronized
• Timestamp continuity is maintained between consecutive segments

Actual behavior:

• Metadata duration is abnormal (e.g., showing 22 hours 33 minutes for 6-second content)
• Large discrepancy between audio and video timestamps (approximately 22 hours difference)
• Bitrates are recorded with abnormally low values

Example ffprobe output:

Input #0, mpegts, from 'stream_001.ts':
  Duration: 22:33:06.22, start: 14281.163922, bitrate: 0 kb/s
  ...
  Stream #0:0[0x100]: Video: h264 (Constrained Baseline) ([27][0][0][0] / 0x001B), yuv420p(progressive), 800x600, 15 fps, 15 tbr, 90k tbn
    start_time=14281.163922
    duration=6.000011
  Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 130 kb/s
    start_time=95461.388522
    duration=5.994667

Possible solution

The timestamp handling in Kurento 7 needs to be examined, particularly during WebRTC to RTP conversion. The issue may be related to the change in timestamp units from seconds to milliseconds that occurred when migrating from Kurento 6 to 7.

A review of the RtpEndpoint implementation, especially the parts handling timestamp and PTS calculation/conversion, may be necessary.

Info about your environment

Pipeline configuration:

• WebRtcEndpoint -> RtpEndpoint (direct connection)
• Or WebRtcEndpoint -> Composite -> RtpEndpoint (connection after composition)

Network configuration:

• All components are located within the same local network
• No STUN or TURN is used (all local connections)

About Kurento Media Server

• Kurento version: 7.2.0
• Server OS: Ubuntu 24.04
• Installation method:
  • [apt-get]
  • [Docker]
  • [AWS CloudFormation]
  • [Built from sources]

About your Application Server

• Programming Language: Scala
• Kurento Client version: 7.2.0

About end-user clients

• Device(s): iPhone
• OS(es): iOS 16.7.8

hls_composite.zip
hls_direct.zip

[slabajo]

Thanks for your so detailed issue description. I think that your assumption of the change from seconds to milliseconds may be relevant, but we need to verify, as we have several applications that integrate WebRTC and RTP and we didn't observe such behaviour on Kurento, 7. Allow us some time to reproduce your case and verify results and also RTP dump.

I need to ask if Kurento is transcoding from WebRTC to RTP or not. is H264 forced on WebRTC, or there may be some transcoding from VP8 to H264? This may also be relevant for the issue.

[y-sugitani]

Thank you for your quick response.

I am forcing H264 in the SdpEndpoint.conf.json configuration:

    "videoCodecs": [
        { "name": "H264/90000" }
    ]

Regarding transcoding:

• When using the Composite, there is transcoding happening from Raw to H264
• When connecting WebRTC directly to RTP, there might not be any transcoding, but I haven't investigated this thoroughly as we don't use direct connections in our production environment

For reference, I ran a similar experiment with Kurento 6, and the generated Duration and start time values were completely normal. This further suggests that the issue is specific to Kurento 7's timestamp handling.

Additionally, I tried using VP9 instead by changing the SDP as follows:

v=0
o=- 0 0 IN IP4 192.168.1.203
s=Streaming Session
c=IN IP4 192.168.1.203
t=0 0
m=video 5004 RTP/AVP 96
a=rtpmap:96 VP9/90000
a=recvonly
m=audio 5006 RTP/AVP 97
a=rtpmap:97 opus/48000/2
a=recvonly

However, when attempting to have FFmpeg transcode from VP9 to H264 and save as MP4, the resulting file still couldn’t be played properly.

[y-sugitani]

I've revisited this issue with assistance from Claude Opus (AI assistant) to conduct a deeper investigation. While I don't fully understand all the technical details, the workaround appears to fix the problem, so I'm sharing these findings.

Investigation Findings

Through analysis of the actual TS files and GStreamer source code, we observed the following:

KMS7 (before workaround):

Video start_pts: 48089430  (534 seconds)
Audio start_pts: 7015977180 (77,955 seconds)
Difference: ~77,421 seconds (21.5 hours!)

KMS6 (working correctly):

Video start_pts: 7058274092 (78,425 seconds)
Audio start_pts: 7058270672 (78,425 seconds)
Difference: ~0 seconds (synchronized)

Hypothesis

Based on GStreamer source code analysis, we suspect that each RTP payloader (audio and video) independently generates a random timestamp-offset value. This happens because rtpsession does not provide timestamp-offset in peer caps during negotiation, causing the payloader to fall back to random values per RFC 3550 recommendations.

In KMS6, audio and video timestamps appear to be synchronized. In KMS7, they are not, resulting in massive PTS differences in the output.

Workaround That Works

We added a pad probe on send_rtp_src pads to reset RTP timestamps by subtracting the first packet's timestamp from all subsequent packets. This makes each stream's timestamps start from 0.

The following patch to kmsbasertpendpoint.c resolved the issue for us:

/* RTP timestamp reset for send path */
typedef struct _RtpTimestampResetData
{
  GMutex mutex;
  guint32 base_timestamp;
  gboolean initialized;
} RtpTimestampResetData;

static RtpTimestampResetData *
rtp_timestamp_reset_data_new ()
{
  RtpTimestampResetData *data = g_slice_new0 (RtpTimestampResetData);
  g_mutex_init (&data->mutex);
  data->base_timestamp = 0;
  data->initialized = FALSE;
  return data;
}

static void
rtp_timestamp_reset_data_destroy (RtpTimestampResetData * data)
{
  g_mutex_clear (&data->mutex);
  g_slice_free (RtpTimestampResetData, data);
}

static void
kms_base_rtp_endpoint_reset_rtp_timestamp (GstBuffer * buffer,
    RtpTimestampResetData * data, GstPad * pad)
{
  GstRTPBuffer rtp_buffer = GST_RTP_BUFFER_INIT;
  guint32 timestamp;

  if (!gst_rtp_buffer_map (buffer, GST_MAP_READWRITE, &rtp_buffer)) {
    return;
  }

  timestamp = gst_rtp_buffer_get_timestamp (&rtp_buffer);

  g_mutex_lock (&data->mutex);

  if (!data->initialized) {
    data->base_timestamp = timestamp;
    data->initialized = TRUE;
    GST_INFO_OBJECT (pad, "RTP timestamp reset: base=%u", data->base_timestamp);
  }

  /* Subtract base to make timestamps start from 0 */
  gst_rtp_buffer_set_timestamp (&rtp_buffer, timestamp - data->base_timestamp);

  g_mutex_unlock (&data->mutex);

  gst_rtp_buffer_unmap (&rtp_buffer);
}

static gboolean
kms_base_rtp_endpoint_reset_buffer_list_foreach (GstBuffer ** buffer,
    guint idx, gpointer user_data)
{
  gpointer *args = user_data;
  RtpTimestampResetData *data = args[0];
  GstPad *pad = args[1];

  *buffer = gst_buffer_make_writable (*buffer);
  kms_base_rtp_endpoint_reset_rtp_timestamp (*buffer, data, pad);

  return TRUE;
}

static GstPadProbeReturn
kms_base_rtp_endpoint_rtp_timestamp_reset_probe (GstPad * pad,
    GstPadProbeInfo * info, gpointer user_data)
{
  RtpTimestampResetData *data = user_data;

  if (info->type & GST_PAD_PROBE_TYPE_BUFFER) {
    GstBuffer *buffer = GST_PAD_PROBE_INFO_BUFFER (info);
    if (buffer != NULL) {
      buffer = gst_buffer_make_writable (buffer);
      GST_PAD_PROBE_INFO_DATA (info) = buffer;
      kms_base_rtp_endpoint_reset_rtp_timestamp (buffer, data, pad);
    }
  } else if (info->type & GST_PAD_PROBE_TYPE_BUFFER_LIST) {
    GstBufferList *buffer_list = GST_PAD_PROBE_INFO_BUFFER_LIST (info);
    if (buffer_list != NULL) {
      gpointer args[2] = { data, pad };
      buffer_list = gst_buffer_list_make_writable (buffer_list);
      GST_PAD_PROBE_INFO_DATA (info) = buffer_list;
      gst_buffer_list_foreach (buffer_list,
          kms_base_rtp_endpoint_reset_buffer_list_foreach, args);
    }
  }

  return GST_PAD_PROBE_OK;
}

And in kms_base_rtp_endpoint_rtpbin_pad_added():

  /* Handle send_rtp_src pads - add timestamp reset probe for outgoing RTP.
   * This ensures RTP timestamps start from 0, which is required for
   * proper playback in downstream receivers like Wowza/Safari. */
  if (g_str_has_prefix (pad_name, RTPBIN_SEND_RTP_SRC)) {
    RtpTimestampResetData *reset_data = rtp_timestamp_reset_data_new ();
    gst_pad_add_probe (pad,
        GST_PAD_PROBE_TYPE_BUFFER | GST_PAD_PROBE_TYPE_BUFFER_LIST,
        kms_base_rtp_endpoint_rtp_timestamp_reset_probe,
        reset_data,
        (GDestroyNotify) rtp_timestamp_reset_data_destroy);
    GST_INFO_OBJECT (self, "Added RTP timestamp reset probe on %s", pad_name);
    added = FALSE;  /* No MEDIA_START signal for send pads */
    goto end;
  }

Results After Workaround

KMS7 (with workaround):

Video start_pts: 13874940 (154.166 seconds)
Audio start_pts: 13872630 (154.140 seconds)
Difference: ~0.026 seconds (synchronized!)

Safari HLS playback now works correctly, including mid-stream join and reload scenarios.

Notes

• I don't fully understand why this specific approach works or what the proper fix should be
• The workaround appears to solve the immediate problem
• There may be a more appropriate solution at the architectural level

Hope this information helps in identifying and fixing the root cause.