Add new consecutiveDroppedFrames callback #1917

khouzam · 2024-11-25T23:35:39Z

This is similar to the droppedFrames callback, but represents the number of consecutive frames that we dropped before rendering a frame or seeking or stopping the renderer.

While we already have a maxConsecutiveDroppedFrame available in the DecoderCounters, this doesn't provide enough visibility into the actual statistics of dropped frames.

If we get 200 dropped frames and a maxConsecutive of 20, we don't know if we dropped 20 frames in a row once and then dropped a single frame 180 times or if we dropped 20 frames 10 times.

We could add some code on our OnDroppedFrames callback to estimate if two calls are for consecutive frames, but that seems very fragile.

There is an open issue currently on how often to notify, and my preference would be to add a new minConsecutiveDroppedFramesToNotify similar to the maxDroppedFramesToNotify but that would only notify if more than X consecutive frames were dropped.

microkatz · 2024-12-03T12:23:47Z

libraries/exoplayer/src/main/java/androidx/media3/exoplayer/analytics/AnalyticsCollector.java

+   * Called to report the number of consecutive frames dropped by the video renderer. Consecutive
+   * dropped frames are reported when a frame is renderered after a the previous consecutive frames
+   * were not rendered optionally, whenever the consecutive dropped frame count is above a specified
+   * threshold whilst the renderer is started.


The javadoc for this method is a bit confusing. Perhaps there were some copy-paste logic errors from the onDroppedFrames javadoc?

With your proposed changes, MediaCodecVideoRenderer will call onConsecutiveDroppedFrames when the renderer successfully renderers a frame after dropping previous frames. Perhaps some developers may subclass MediaCodecVideoRenderer and DefaultRenderersFactory in order to add a threshold for reporting this value as well.

The comment has been fixed and should be much clearer as to the behavior.

microkatz · 2024-12-03T12:57:25Z

libraries/exoplayer/src/main/java/androidx/media3/exoplayer/video/MediaCodecVideoRenderer.java

@@ -1945,6 +1947,17 @@ private void maybeNotifyDroppedFrames() {
    }
  }

+  private void maybeNotifyConsecutiveDroppedFrames() {
+    // TODO: Figure out the notification threshold for consecutive dropped frames.
+    if (consecutiveDroppedFrameCount > 0) {


Can you add a threshold value, maxConsecutiveDroppedFramesToNotify? This would include adding an addition constructor for MediaCodecVideoRenderer and a default value in DefaultRenderersFactory.

Our team will discuss the best initial value but for now you can set it the same as MAX_DROPPED_VIDEO_FRAME_COUNT_TO_NOTIFY which is 50.

Tying the callback to a threshold will require applications that care about this metric to set a reasonable threshold for themselves. If an application does not care about this metric then the event will not be dispatched as often.

Added a new MIN_CONSECUTIVE_DROPPED_VIDEO_FRAME_COUNT_TO_NOTIFY = 5 value in the Default Renderer. Only getting notified when 50 consecutive frames have been dropped seems very high. But I'll defer the final value to the team.

I've updated all the constructors to take in the value, but that seems like a breaking change.

microkatz · 2024-12-03T13:05:13Z

...ries/exoplayer/src/main/java/androidx/media3/exoplayer/video/VideoRendererEventListener.java

+        handler.post(
+            () ->
+                castNonNull(listener)
+                    .onConsecutiveDroppedFrames(consecutiveDroppedFrameCount, elapsedMs));


Can you add a test to MediaCodecVideoRendererTest to make sure that provided a very late buffer that is dropped that this event is invoked? You can look at MediaCodecVideoRendererTest::render_withLateBuffer_dropsBuffer. May also be helpful to have unit tests for the event testing the output for if dropped frames occur prior to any successful renders and between successful renders.

Added tests covering both the API being invoked and only being invoked when there are enough frame drops.

microkatz · 2024-12-03T13:06:52Z

libraries/exoplayer/src/main/java/androidx/media3/exoplayer/analytics/AnalyticsListener.java

@@ -1244,6 +1248,18 @@ default void onVideoInputFormatChanged(
  @UnstableApi
  default void onDroppedVideoFrames(EventTime eventTime, int droppedFrames, long elapsedMs) {}

+  /**
+   * Called after consecutive video frames have been dropped.


Note that this event is called after an 'end' to the run of consecutive dropped frames.

microkatz · 2024-12-03T13:11:52Z

libraries/exoplayer/src/main/java/androidx/media3/exoplayer/video/MediaCodecVideoRenderer.java

+  private void maybeNotifyConsecutiveDroppedFrames() {
+    // TODO: Figure out the notification threshold for consecutive dropped frames.
+    if (consecutiveDroppedFrameCount > 0) {
+      long elapsed = getClock().elapsedRealtime() - consecutiveDroppedFrameAccumulationStartTimeMs ;


nit: Include unit in variable name, elapsedMs.

This is similar to the `droppedFrames` callback, but represents the number of consecutive frames that we dropped before rendering a frame or seeking or stopping the renderer. While we already have a `maxConsecutiveDroppedFrame` available in the `DecoderCounters`, this doesn't provide enough visibility into the actual statistics of dropped frames. If we get 200 dropped frames and a `maxConsecutive` of 20, we don't know if we dropped 20 frames in a row once and then dropped a single frame 180 times or if we dropped 20 frames 10 times. We could add some code on our `OnDroppedFrames` callback to estimate if two calls are for consecutive frames, but that seems very fragile. Specifying when to invoke the callback is controlled by `minConsecutiveDroppedFramesToNotify` similar to the `maxDroppedFramesToNotify` but that would only notify if more than X consecutive frames were dropped. Adding support for both `MediaCodecVideoRenderer` and `DecoderVideoRenderer`.

colinkho · 2025-05-22T23:08:59Z

Hi @microkatz @tonihei @icbaker , is this ready to be pushed upstream? I see Gilles has addressed most comments

microkatz · 2025-06-05T21:43:19Z

@khouzam

Apologies for the delay!

I was holding off from additional comments and edits as we instituted a new Builder pattern into MediaCodecVideoRenderer rather than adding new constructors for all these new variables/thresholds etc. This PR then got lost in the weeds after that was completed here, decfb9b.

There are a few requests for the PR. First, would you be able to rebase to the tip of main and transition to a Builder parameter for your consecutiveDroppedFrames threshold?

Secondly, in discussions with the team, it was thought better if we could still have a single listener event for dropped frames rather than two. The idea would be to add an additional parameter to onDroppedFrames for maxConsecutive. Currently, onDroppedFrames may be invoked when the dropped frames threshold is reached. This would be changed so that the invocation would occur where you are currently checking for consecutive frame drops notification. At onStopped, onDisabled, and after a successful frame render.

Would this suggestion still work to provide you the granularity of analytics you need?

Please reach out if you have additional questions.

khouzam · 2025-06-06T19:30:39Z

Thanks for the feedback @microkatz, a single listener would also work, as long as we make sure to invoke it appropriately. We'll update the pull request with a rebase and the changes.

icbaker requested a review from microkatz November 26, 2024 09:11

icbaker assigned microkatz Nov 26, 2024

microkatz reviewed Dec 3, 2024

View reviewed changes

khouzam force-pushed the consecutiveDroppedFrames branch 2 times, most recently from 4ff5889 to a20a1d8 Compare December 4, 2024 23:06

khouzam added 2 commits January 22, 2025 11:49

Merge remote-tracking branch 'origin/main' into consecutiveDroppedFrames

b3a6365

khouzam force-pushed the consecutiveDroppedFrames branch from a20a1d8 to b3a6365 Compare January 22, 2025 19:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add new consecutiveDroppedFrames callback #1917

Add new consecutiveDroppedFrames callback #1917

Uh oh!

khouzam commented Nov 25, 2024

Uh oh!

microkatz Dec 3, 2024

Uh oh!

khouzam Dec 4, 2024

Uh oh!

microkatz Dec 3, 2024

Uh oh!

khouzam Dec 4, 2024

Uh oh!

microkatz Dec 3, 2024

Uh oh!

khouzam Dec 4, 2024

Uh oh!

microkatz Dec 3, 2024

Uh oh!

microkatz Dec 3, 2024

Uh oh!

colinkho commented May 22, 2025

Uh oh!

microkatz commented Jun 5, 2025

Uh oh!

khouzam commented Jun 6, 2025

Uh oh!

Uh oh!

Add new consecutiveDroppedFrames callback #1917

Are you sure you want to change the base?

Add new consecutiveDroppedFrames callback #1917

Uh oh!

Conversation

khouzam commented Nov 25, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

colinkho commented May 22, 2025

Uh oh!

microkatz commented Jun 5, 2025

Uh oh!

khouzam commented Jun 6, 2025

Uh oh!

Uh oh!