[RSDK-6469] Set x264 bitrate based on resolution and fps #4769

seanavery · 2025-02-03T17:52:38Z

Description

This PR gives a more accurate bitrate estimation to our libx264 pipeline based on resolution and framerate. The idea here is that smaller streams will use less bandwidth and we will get better quality out of larger streams.

Removes hardcoded 3.2 Mbps bitrate.
Adds look up for framerate from Properties and if it is not found default to 30.
Calculates bitrate using a reasonable 0.15 bits per pixel ratio.
Adds minimum bitrate to avoid buffering issues with small frame sizes.
Adds a maximum bitrate that is safely over 20fps 4k stream size.

Examples bitrates:

640p at 30 fps: 640*480*30*0.15 - > 1.382400 Mbps
1080p at 30fps : 1920*1080*30*0.15 -> 9.331200 Mbps
4k at 20fps: 3840*2160*20*0.15 -> 24.883200 Mbps

Tests

add unit test for bitrate calculation ✅
webcam
- picks up framerate from properties ✅
- suitable quality across multiple resolutions ✅

Screen.Recording.2025-02-25.at.2.41.22.PM.mov

viamrtsp
- suitable quality at 2k and 4k resolutions ✅
dyanmic resolution
- suitable quality at extremely small frame sizes ✅

seanavery · 2025-02-03T18:03:24Z

components/camera/videosource/webcam.go

@@ -622,6 +622,11 @@ func (c *webcam) Properties(ctx context.Context) (camera.Properties, error) {
 	if err != nil {
 		return camera.Properties{}, err
 	}
+
+	if c.conf.FrameRate > 0 {
+		props.FrameRate = c.conf.FrameRate


Moved framerate props setter outside of conditional IntrinsicParams == nil block.
!!! Need to rebase here with latest webcam for same fix. Fixed in latest. ✅

randhid · 2025-02-03T19:00:54Z

gostream/codec/x264/utils.go

+// calcBitrateFromResolution calculates the bitrate based on the given resolution and framerate.
+func calcBitrateFromResolution(width, height int, framerate float32) int {
+	bitrate := int(float32(width) * float32(height) * framerate * encodeCompressionRatio)
+	if bitrate < minBitrate {


Suggested change

if bitrate < minBitrate {

// this accounts for zero bitrates too

if bitrate < minBitrate {

Sounds good, added comment.

randhid · 2025-02-03T19:02:21Z

robot/web/stream/server.go

 		// We walk the updated set of `videoSources` and ensure all of the sources are "created" and
 		// "started".
 		config := gostream.StreamConfig{
 			Name:                name,
 			VideoEncoderFactory: server.streamConfig.VideoEncoderFactory,
+			TargetFrameRate:     framerate,


What happens if this is 0? Can we add a test?

encoder pipeline will default to 30fps in bitrate calculation (the same as the key-frame interval) if it cant pick up framerate.

Will see about adding a test for this.

hexbabe · 2025-02-03T19:14:37Z

gostream/codec/x264/utils.go

+
+// calcBitrateFromResolution calculates the bitrate based on the given resolution and framerate.
+func calcBitrateFromResolution(width, height int, framerate float32) int {
+	bitrate := int(float32(width) * float32(height) * framerate * encodeCompressionRatio)


Does bitrate scale linearly with width and height? Apparently the answer is complicated https://www.reddit.com/r/explainlikeimfive/comments/hlrwcu/eli5_what_is_the_relationship_between_bit_rate/

I am currently scaling the bitrate linearly with the number of pixels.

Interesting graphs in the thread that show logarithmic quality gains with increasing bitrate for a given size/fps. Will try to verify that we are hitting these sweet-spots. The amount of scene change also comes into play here so it gets quite complicated.

If you look at the example bitrates in the description we seem to be pretty bang on to the recommended settings linked in the jira ticket: https://support.google.com/youtube/answer/2853702?hl=en

hexbabe · 2025-02-03T19:18:29Z

robot/web/stream/server.go

@@ -507,11 +507,16 @@ func (server *Server) AddNewStreams(ctx context.Context) error {
 			server.logger.Warn("video streaming not supported on Windows yet")
 			break
 		}
+		framerate, err := server.getFramerateFromCamera(name)


will this new logic break streaming for cameras that don't provide framerate in getProperties?

Will add comment here -- encoder pipeline will default to 30fps in bitrate calculation (the same as the key-frame interval) if it cant pick up framerate.

seanavery · 2025-02-24T16:10:46Z

@randhid Do you think we are ok to merge this?

randhid

Can we :

add tests for this logic
Verify that adding and removing streams from multiple sources works still and report the results of the manual testing? I'd like a video.

randhid · 2025-02-24T16:26:40Z

gostream/codec/x264/utils.go

+
+// calcBitrateFromResolution calculates the bitrate based on the given resolution and framerate.
+func calcBitrateFromResolution(width, height int, framerate float32) int {
+	bitrate := int(float32(width) * float32(height) * framerate * encodeCompressionRatio)


This could result in a float? Does that break the stream sever?

The int cast will remove decimal portion effectively rounding down to nearest int. Changed to round up instead which seems to make more sense.

randhid · 2025-02-24T16:27:58Z

gostream/codec/x264/utils.go

@@ -27,3 +38,16 @@ func (f *factory) New(width, height, keyFrameInterval int, logger logging.Logger
 func (f *factory) MIMEType() string {
 	return "video/H264"
 }
+


Can't comment above - but I just realized - if the initial resolution is odd - x264 could break could we add a check for the initial resolution not being odd as well?

Good call -- added dimension check to NewEncoder.

randhid · 2025-02-25T18:46:17Z

gostream/codec/x264/encoder.go

+	if width%2 != 0 || height%2 != 0 {
+		return nil, errors.New("x264 encoder does not support odd dimensions")
+	}


If an initial camera resoution is odd - could we change the resolution the same way that we do when we're setting stream options?

Or more explicitly tell the user in this error to return an image that is even dimensions? I think I prefer this option.

Because where this is coming from is someone badly encoding a random resolution image.

Ok, improved error message here to inform user to provide even dim frames.

If an initial camera resoution is odd - could we change the resolution the same way that we do when we're setting stream options?

We could do something fancy like this but not sure if it is worth it given that realistically all cameras should be providing even dimension frames. The only time I have seen this is when fake camera allowed odd dimensions.

So pete made a camera where he cropped, re-saturated and transformed a bunch of things from an ffmpeg camera and ended up with odd resolutions in the new camera. Remember the return of bytes is really up to the implementer and you can do wild things if you're doing really custom things.

Sounsd good. This check should still work in such cases because if there is a size change we re-initialize the encoder in the stream loop and the new resolution (if odd) should hit this check.

seanavery

@randhid Added unit tests for bitrate calculation and put up manual test videos in description.

randhid · 2025-02-25T20:13:25Z

Can you test with viamrtsp?

seanavery · 2025-02-25T20:28:21Z

@randhid

Can you test with viamrtsp?

Yep, tested quite a bit with viamrtsp -- see description.

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Feb 3, 2025

seanavery commented Feb 3, 2025

View reviewed changes

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Feb 3, 2025

seanavery added 6 commits February 3, 2025 13:32

Handle bitrate calc in gostream

7eaf687

Add fps lookup to stream configs

1780a0c

Remove debug logs

11f81ae

Clean

3573204

Lint

4bbc895

Add comment

e789a17

seanavery force-pushed the RSDK-6469 branch from 164905f to e789a17 Compare February 3, 2025 18:33

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Feb 3, 2025

seanavery requested review from hexbabe and randhid February 3, 2025 18:54

randhid requested changes Feb 3, 2025

View reviewed changes

hexbabe approved these changes Feb 3, 2025

View reviewed changes

hexbabe self-requested a review February 3, 2025 19:18

hexbabe approved these changes Feb 3, 2025

View reviewed changes

Add comment about framerate fetch failure case

c11fb11

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Feb 3, 2025

Add comment about 0 bitrate handling

86affca

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Feb 3, 2025

seanavery requested a review from randhid February 19, 2025 18:51

randhid requested changes Feb 24, 2025

View reviewed changes

Round up

9f8c7f7