[RSDK-9132] Add (Get)Image to the camera interface #4487

hexbabe · 2024-10-24T19:00:39Z

Overview

https://viam.atlassian.net/browse/RSDK-9132

This PR does not remove Stream from the Camera interface or any gostream logic. It just wraps it all in better English that corresponds with the gRPC method GetImage. Mostly just adding syntactic sugar over stream.Next/ReadImage using a new Image method (for existing builtins that use gostream), and modifying the camera server and client appropriately to utilize Image.

My hopes are that this encourages people to never be tempted to use Stream in modules or even VideoReaderFunc. We will eventually removeStream from the base Camera interface

It will also provide a significant performance boost to the ReadImage data collector since we no longer have to transcode the old image.Image output into bytes anymore in collector.go.

Summary of changes

Add Image to camera interface
Use Image in camera/client.go, camera/server.go and data collector
Use Image in builtins (webcam, replaypcd, ffmpeg, fake etc.) instead of Stream wrappers in preparation for removing Stream entirely from the camera interface

Note

Will this break cli module generator? No

Manual Tests

Did this list of manual tests already twice at different points in development, but should re-test after these code reviews.

viamrtsp h264 with and without passthrough
Webcam
Webcam hot swap (reconfigure/rebuild due to switching sources)
Transform pipeline on webcam (rotate and crop including new crop features)
ffmpeg behavior with video0 is on par with the main branch (broken?? yes made ticket to track)
ffmpeg works for a fake mediamtx+ffmpeg rtsp stream though! No hang and stream looks great
Fake camera looks good with and without rtp passthrough toggled
OAK-FFC-3P

Also built the viamrtsp module with local replacement off this branch and the stream is working for both rtp passthrough true and false, indicating that videosourcewrapped modules shouldn't be affected by this change.

Profile for performance improvement in data collection

If the PDFs are insanely blurry, try downloading and opening with system viewer

Link

The big thing that Nick noted was alarming was the staggering memory used to alloc image NewYCbCr. This is no longer the case on this feature branch.

Config used for pprof

{
  "components": [
    {
      "name": "nickcam-big-green-1",
      "namespace": "rdk",
      "type": "camera",
      "model": "ncs:camera:nickcam",
      "attributes": {
        "big": true,
        "color": "green",
        "image_type": "jpeg"
      },
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "method": "ReadImage",
                "capture_frequency_hz": 60,
                "additional_params": {},
                "capture_queue_size": 0
              }
            ]
          }
        }
      ]
    },
    {
      "name": "nickcam-big-green-2",
      "namespace": "rdk",
      "type": "camera",
      "model": "ncs:camera:nickcam",
      "attributes": {
        "color": "green",
        "image_type": "jpeg",
        "big": true
      },
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "method": "ReadImage",
                "capture_frequency_hz": 60,
                "additional_params": {},
                "capture_queue_size": 0
              }
            ]
          }
        }
      ]
    },
    {
      "name": "nickcam-big-green-3",
      "namespace": "rdk",
      "type": "camera",
      "model": "ncs:camera:nickcam",
      "attributes": {
        "image_type": "jpeg",
        "big": true,
        "color": "green"
      },
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "method": "ReadImage",
                "capture_frequency_hz": 60,
                "additional_params": {},
                "capture_queue_size": 0
              }
            ]
          }
        }
      ]
    }
  ],
  "services": [
    {
      "name": "data_manager-1",
      "namespace": "rdk",
      "type": "data_manager",
      "attributes": {
        "sync_disabled": true,
        "capture_disabled": false
      }
    }
  ],
  "modules": [
    {
      "type": "registry",
      "name": "ncs_nickcam",
      "module_id": "ncs:nickcam",
      "version": "latest"
    }
  ]
}

Comparative performance from an operator's perspective

Config used

{
  "components": [
    {
      "name": "rtsp-1",
      "namespace": "rdk",
      "type": "camera",
      "model": "viam:viamrtsp:rtsp",
      "attributes": {
        "rtsp_address": "rtsp://10.1.12.67:554"
      },
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "method": "ReadImage",
                "capture_frequency_hz": 60,
                "additional_params": {},
                "capture_queue_size": 0
              }
            ]
          }
        }
      ]
    },
    {
      "name": "rtsp-2",
      "namespace": "rdk",
      "type": "camera",
      "model": "viam:viamrtsp:rtsp",
      "attributes": {
        "rtsp_address": "rtsp://10.1.12.67:554"
      },
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "method": "ReadImage",
                "capture_frequency_hz": 60,
                "additional_params": {},
                "capture_queue_size": 0
              }
            ]
          }
        }
      ]
    },
    {
      "name": "rtsp-3",
      "namespace": "rdk",
      "type": "camera",
      "model": "viam:viamrtsp:rtsp",
      "attributes": {
        "rtsp_address": "rtsp://10.1.12.67:554"
      },
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "method": "ReadImage",
                "capture_frequency_hz": 60,
                "additional_params": {},
                "capture_queue_size": 0
              }
            ]
          }
        }
      ]
    },
    {
      "name": "rtsp-4",
      "namespace": "rdk",
      "type": "camera",
      "model": "viam:viamrtsp:rtsp",
      "attributes": {
        "rtsp_address": "rtsp://10.1.12.67:554"
      },
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "method": "ReadImage",
                "capture_frequency_hz": 60,
                "additional_params": {},
                "capture_queue_size": 0
              }
            ]
          }
        }
      ]
    },
    {
      "name": "rtsp-5",
      "namespace": "rdk",
      "type": "camera",
      "model": "viam:viamrtsp:rtsp",
      "attributes": {
        "rtsp_address": "rtsp://10.1.12.67:554"
      },
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "method": "ReadImage",
                "capture_frequency_hz": 60,
                "additional_params": {},
                "capture_queue_size": 0
              }
            ]
          }
        }
      ]
    }
  ],
  "services": [
    {
      "name": "data_manager-1",
      "namespace": "rdk",
      "type": "data_manager",
      "attributes": {
        "sync_disabled": true,
        "capture_disabled": false
      }
    }
  ],
  "modules": [
    {
      "type": "registry",
      "name": "viam_viamrtsp",
      "module_id": "viam:viamrtsp",
      "version": "0.2.0-rc4",
      "disabled": false
    }
  ]
}

v0.50.0 branch (RPI4):

Feature branch (RPI4):

v0.50.0 (Orin Nano):

Feature branch (Orin Nano):

RPI4 memory improvement: I noticed that off the feature branch, memory stabilized around 1.9gb, while on the main branch, memory stabilized around 2.5gb

RPI4 CPU improvement: I learned how to filter for processes in this comparison. It's quite clear that the CPU usage is lower as all figures are lower across the board for the feature branch.

Orin Nano CPU improvement: Though the core usage was about the same between main and feature, I noticed that on the latest release, the viam-server processes were always taking the most CPU. Off the feature branch, viamrtsp were always on top.

I did not notice memory improvements on the Orin Nano. May be missing something. Operator skill issue perhaps

github-actions · 2024-10-24T19:00:54Z

Warning your change may break code samples. If your change modifies any of the following functions please contact @viamrobotics/fleet-management. Thanks!

component	function
base	IsMoving
board	GPIOPinByName
camera	Properties
encoder	Properties
motor	IsMoving
sensor	Readings
servo	Position
arm	EndPosition
audio	MediaProperties
gantry	Lengths
gripper	IsMoving
input_controller	Controls
movement_sensor	LinearAcceleration
power_sensor	Power
pose_tracker	Poses
motion	GetPose
vision	GetProperties

hexbabe · 2024-10-25T18:59:35Z

I'm gonna be out next week but this is ready for a first pass review. Also lmk if we should add Nick or/(and?) Dan to this review. Perhaps for future reviews that are a bit more invasive?

randhid

Since we're breaking go modules by adding another method to the interface, we should make that method follow 1:1 the inputs and returns of the proto api.

This will mean bookkeeping the release of each image stream,should you choose to use ReadImage. This will definitely go away once we get the image from the underlying comms connection like we do in the intel realsense and oak-d camera, I don't know how intense that is.

If changing the returns from GetImage to just be 1:1 the API type cause more problems in changing code outside of camera, we can focus on replacements in the camera packages, and let datamanger and vision packages use Stream and Next for this pr, but I think you're nearly there.

components/camera/camera.go

components/camera/camera_test.go

components/camera/client.go

components/camera/replaypcd/replaypcd.go

components/camera/server.go

components/camera/transformpipeline/mods.go

components/camera/videosource/webcam.go

components/camera/camera.go

seanavery · 2024-11-01T19:48:45Z

components/camera/client.go

@@ -228,6 +228,10 @@ func (c *client) Stream(
 	return stream, nil
 }

+func (c *client) GetImage(ctx context.Context) (image.Image, func(), error) {
+	return c.Read(ctx)


Could we Read and handle release so we do not need to include release func in returns? (perhaps that would make sense in the videosourcewrapper layer).
As mentioned, this would be a larger departure from Stream.Next which assuming will cause major problems.

I'm gonna try to get rid of release in camera components and see what happens

Looks like we just need to move the rimage.EncodeImage step one level in i.e. outBytes should come out of the Image call instead of being handled in the data collector or camera server/client. Currently release is called in the collector/server/client, so we should just move it into the Image implementation. Same logic, just handled more nested in our abstraction.

I guess future module writers should get used to using rimage to encode their output i.e. read from the source bytes and output a newly constructed []byte result? Does that sound okay to everyone?

Looks like we just need to move the rimage.EncodeImage step one level in i.e. outBytes should come out of the Image call instead of being handled in the data collector or camera server/client.

In the client, I think that makes sense to return rimage to match the python sdk functionality -- avoid unnecessary decodes if you just want bytes.

I guess future module writers should get used to using rimage to encode their output i.e. read from the source bytes and output a newly constructed []byte result? Does that sound okay to everyone?

Would this be similar to viamimage in the python sdk, i think that works pretty well.

Looks like the server is already handling release and encoding for the caller.
Are you also suggesting removing encoding step in server GetImage?

This is my WIP in server.go

switch castedCam := cam.(type) { case ReadImager: // RSDK-8663: If available, call a method that reads exactly one image. The default // `ReadImage` implementation will otherwise create a gostream `Stream`, call `Next` and // `Close` the stream. However, between `Next` and `Close`, the stream may have pulled a // second image from the underlying camera. This is particularly noticeable on camera // clients. Where a second `GetImage` request can be processed/returned over the // network. Just to be discarded. // RSDK-9132(sean yu): In addition to what Dan said above, ReadImager is important // for camera components that rely on the `release` functionality provided by gostream's `Read` // such as viamrtsp. // (check that this comment is 100% true before code review then delete this paranthetical statement) img, release, err := castedCam.Read(ctx) if err != nil { return nil, err } defer func() { if release != nil { release() } }() actualMIME, _ := utils.CheckLazyMIMEType(req.MimeType) resp.MimeType = actualMIME outBytes, err := rimage.EncodeImage(ctx, img, req.MimeType) if err != nil { return nil, err } resp.Image = outBytes default: imgBytes, mimeType, err := cam.Image(ctx, req.MimeType, ext) if err != nil { return nil, err } actualMIME, _ := utils.CheckLazyMIMEType(mimeType) resp.MimeType = actualMIME resp.Image = imgBytes }

So I think yes, in the default case we don't encode anymore since the return type is now bytes

Ok cool I like that.

I am little confused on why we still need the ReadImager path here. Shouldnt the camera interface now always have Image defined so we can just use that?

My thinking here is since viamrtsp uses the VideoReaderFunc and Read's release functionality to keep track of pool frames in the frame allocation optimization flow, for the server.go that serves viamrtsp, we need to be able to call release()

I think we could refactor viamrtsp though to just copy out the bytes on return early and use the new .Image pathway... I'm down to remove ReadImager entirely as long as we make it a high priority to refactor viamrtsp to use .Image and []byte

I see, interesting. I think it makes sense to leave in for now for viamrtsp compatibility.

I think we could refactor viamrtsp though to just copy out the bytes on return early and use the new .Image pathway... I'm down to remove ReadImager entirely as long as we make it a high priority to refactor viamrtsp to use .Image and []byte

Yep the whole point to having a memory manager was the issue with passing in a pointer to the avframe in VideoReaderFunc return causing segfaults with the encoding in the server.

Since we are removing encoding from the server and relying on the user to handle this, we should eventually be able to use Image pathway and simplify things quite a bit : )

All our go cameras will need a refactor for the full fix of this interface, I do not anticipate that we will have the best version of this code and go modules after one pr. Eventually we will get rid of Stream and Next completely. The problem is those videosourcewrapper helpers are all over the place in go modules since they're exported convenience functions.

Always export code conscientiously. But we're breaking this interface anyway, so we'll deal with the following breaks.

Okay 👍 will remove ReadImager in this PR

components/camera/server.go

components/camera/transformpipeline/classifier.go

seanavery

I am wondering if it makes sense to do a full removal of Read from client & server, ReadImage, and ReadImager here instead of just aliasing in those cases.

randhid · 2024-11-01T20:50:12Z

I am wondering if it makes sense to do a full removal of Read from client & server, ReadImage, and ReadImager here instead of just aliasing in those cases.

Yes.

seanavery

Should we replace the cemeraservice GetImage call in RenderFrame with Image?

func (s *serviceServer) RenderFrame(
	ctx context.Context,
	req *pb.RenderFrameRequest,
) (*httpbody.HttpBody, error) {
	ctx, span := trace.StartSpan(ctx, "camera::server::RenderFrame")
	defer span.End()
	if req.MimeType == "" {
		req.MimeType = utils.MimeTypeJPEG // default rendering
	}
	resp, err := s.GetImage(ctx, (*pb.GetImageRequest)(req))

hexbabe · 2024-11-06T17:22:50Z

Should we replace the cemeraservice GetImage call in RenderFrame with Image?

func (s *serviceServer) RenderFrame(
	ctx context.Context,
	req *pb.RenderFrameRequest,
) (*httpbody.HttpBody, error) {
	ctx, span := trace.StartSpan(ctx, "camera::server::RenderFrame")
	defer span.End()
	if req.MimeType == "" {
		req.MimeType = utils.MimeTypeJPEG // default rendering
	}
	resp, err := s.GetImage(ctx, (*pb.GetImageRequest)(req))

serviceServers use the gRPC named version, so GetImage is right

…t case in client_test.go

components/camera/client_test.go

nicksanford

LGTM Great work Sean!

I also confirmed that filtered camera still works as expected.

nicksanford · 2024-12-02T22:54:08Z

LGTM Great work Sean!

I also confirmed that filtered camera still works as expected.

Also, the end to end of syncing camera.ReadImage (for modules).

I did not test the transform camera.

Co-authored-by: nicksanford <[email protected]>

hexbabe · 2024-12-03T14:55:21Z

Cool. I'm gonna run it through my own manual test gauntlet and then merge before EOD

Co-authored-by: nicksanford <[email protected]>

Init craziness

a643d22

viambot added the safe to test This pull request is marked safe to test from a trusted zone label Oct 24, 2024

Use camera pkg scoped ReadImage in webcam

41cb592

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Oct 25, 2024

hexbabe marked this pull request as ready for review October 25, 2024 18:56

hexbabe requested review from randhid and seanavery October 25, 2024 18:58

randhid requested changes Oct 31, 2024

View reviewed changes

seanavery reviewed Nov 1, 2024

View reviewed changes

Merge branch 'main' into RSDK-9132

f6e3d69

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 4, 2024

seanavery reviewed Nov 5, 2024

View reviewed changes

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 6, 2024

hexbabe requested review from ethanlookpotts, micheal-parks, DTCurrie and zaporter-work as code owners November 6, 2024 17:59

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 6, 2024

Use agreed upon Image signature

d6439dd

hexbabe force-pushed the RSDK-9132 branch from f90689b to d6439dd Compare November 6, 2024 18:10

viambot removed the safe to test This pull request is marked safe to test from a trusted zone label Nov 6, 2024

viambot added the safe to test This pull request is marked safe to test from a trusted zone label Dec 2, 2024

Add actual image content check for "lazily decode from ReadImage" tes…

cdd544f

…t case in client_test.go

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 2, 2024

nicksanford reviewed Dec 2, 2024

View reviewed changes

components/camera/client_test.go Show resolved Hide resolved

components/camera/client_test.go Show resolved Hide resolved

nicksanford approved these changes Dec 2, 2024

View reviewed changes

Update components/camera/client_test.go

1602152

Co-authored-by: nicksanford <[email protected]>