Buffer visualization

[kixelated]

It would be nice (in the stats page?) to visualize the current buffer size. I was thinking horizontal bars with the left side being 0 and the right side being the target latency.

MoQ is interesting because segments/groups can arrive out of order. Instead of a single buffered amount, we need to support multiple ranges. Audio and video need to be represented separately too.

[kixelated]

Actually, it would be really cool to revamp the latency slider so it is aligned with this buffer visualization. I think that's better than putting it on the stats page.

[jdreetz]

MoQ is interesting because segments/groups can arrive out of order. Instead of a single buffered amount, we need to support multiple ranges. Audio and video need to be represented separately too.

@kixelated do you have any recommendations on how to measure this? Right now we're tracking the buffer size (in the Stats UI) as how long we "sleep" for between rendering frames.

I've been trying to measure the buffer size by looking at the min/max timestamps of frames in this.#groups.frames inside the frame Consumer, but that timespan is always zero, unless I do something like wait for the TransformStream writer to be ready before we decode the frame, but adding that wait caused the buffer size to start to creep up over time.

What did you have in mind for measuring this? If we use the "sleep" time, that's usually only about 40ms, even if I change the latency slider to a higher value, so I don't think is giving us something representative of the latency value, or is it?

Also, I'm not sure if I should be testing this differently, like adding an artificial delay somehow to simulate poor network conditions?

[kixelated]

MoQ is interesting because segments/groups can arrive out of order. Instead of a single buffered amount, we need to support multiple ranges. Audio and video need to be represented separately too.

@kixelated do you have any recommendations on how to measure this? Right now we're tracking the buffer size (in the Stats UI) as how long we "sleep" for between rendering frames.

I've been trying to measure the buffer size by looking at the min/max timestamps of frames in this.#groups.frames inside the frame Consumer, but that timespan is always zero, unless I do something like wait for the TransformStream writer to be ready before we decode the frame, but adding that wait caused the buffer size to start to creep up over time.

What did you have in mind for measuring this? If we use the "sleep" time, that's usually only about 40ms, even if I change the latency slider to a higher value, so I don't think is giving us something representative of the latency value, or is it?

Also, I'm not sure if I should be testing this differently, like adding an artificial delay somehow to simulate poor network conditions?

Yeah it's a bit tricky.

First off, I'm not actually sure if the TransformStream is needed. It doesn't seem to... do anything? IDK b-frames always confuse me and I don't fully understand how they operate with VideoDecoder. Removing it would probably make your life easier.

Second off, yeah you'll have to fake network congestion somehow. Maybe add a setTimeout somewhere in the frame reading logic to add some extra delay or something. The proper solution would be full blown netem.

k so you're right that sleep will be a single frame's length almost all of the time. That's because we already did a long sleep to build up the latency. Basically how it works is that this.#reference is the earliest received timestamp ever (relative to the current time). We linearly extrapolate each subsequent timestamp to figure out when it should be rendered.

This function doesn't exist but it should work:

// Given a timestamp, returns when it should be rendered (in `performance.now()` units)
renderTime(timestamp: Time.Micros): DOMHighResTimestamp {
	return this.#reference + (frame.timestamp / 1000) + this.latency.peek();
}

And yeah, you may have frames stuck inside of the Frame.Consumer. It tries to return groups in order, buffering them if needed if there are out of order groups. It returns the next frame from the current group on demand. I think this is where you will do most of the instrumentation, recording the min/max timestamp within each group and any gaps between them. You can then convert those into wall clock timestamps and factor in the ongoing sleep (to account for the most recent frame).

Also, this.#reference needs to be hoisted to operate at the Broadcast level if you're feeling brave. Currently it's computed on a per-track basis but that won't provide A/V sync. I was going to split it out into its own Sync class that is shared between the components.

[kixelated]

And lemme know if you're having trouble, I can make the API to expose the buffer times. Frame.Consumer is a mess.