Driver crashes if camera disconnects and doesn't recover after reconnect

[sausais]

Description
When using the spinnaker_camera_driver, the driver crashes entirely if the FLIR camera is disconnected even for a very short moment (e.g. 100 ms).
After this short disconnect, the driver throws a Spinnaker::Exception, shuts down, and does not reconnect or recover when the camera is plugged back in. From that point on, the camera stays dead until the entire ROS node is manually restarted.

Expected Behavior

Automatically reconnect and resume streaming when the camera is available again

I manually added Respawn=True, so it starts again. It started but didn't get any frames.
If I restart container it works perfectly. But i need it to reconnect on the go. after a crash

[camera_driver_node-9] [INFO] [1749728638.336545463] [18259964]: rate [Hz] in   1.00 out   1.00 drop   0%
[camera_driver_node-9] [INFO] [1749728643.336546638] [18259964]: rate [Hz] in   1.00 out   1.00 drop   0%
[camera_driver_node-9] WARNING: acquisition timeout, restarting!
[camera_driver_node-9] terminate called after throwing an instance of 'Spinnaker::Exception'
[camera_driver_node-9]   what():  Spinnaker: Could not start acquisition. Please try reconnecting the device. [-1006]
[ERROR] [camera_driver_node-9]: process has died [pid 51, exit code -6, cmd '/opt/ros/jazzy/lib/spinnaker_camera_driver/camera_driver_node --ros-args -r __node:=flir_camera --params-file /tmp/launch_params_bw02_hk_ --params-file /tmp/launch_params_ukw7pyz8 -r ~/control:=/exposure_control/control -r /flir_camera/image_raw:=/image -r /flir_camera/camera_info:=/camera_info'].
[INFO] [camera_driver_node-9]: process started with pid [313]
[camera_driver_node-9] [INFO] [1749728651.220473903] [18259964]: reading ros parameters for camera with serial: 18259964
[camera_driver_node-9] [INFO] [1749728651.220600923] [18259964]: adjusting time stamps!
[camera_driver_node-9] [INFO] [1749728651.220855830] [18259964]: parameter definitions file: /opt/ros/jazzy/share/spinnaker_camera_driver/config/blackfly_s.yaml
[camera_driver_node-9] [INFO] [1749728651.224242236] [flir_camera]: camera calibration URL: file:///xxx/install/rosbag_recorder/share/rosbag_recorder/params/camera_info.yaml
[camera_driver_node-9] [INFO] [1749728653.021675033] [18259964]: using spinnaker lib version: 4.0.0.116
[camera_driver_node-9] [WARN] [1749728653.021980068] [18259964]: no camera found with serial: 18259964 on try # 1
[camera_driver_node-9] [WARN] [1749728654.022867705] [18259964]: no camera found with serial: 18259964 on try # 2
[camera_driver_node-9] [WARN] [1749728655.023916285] [18259964]: no camera found with serial: 18259964 on try # 3
[camera_driver_node-9] [WARN] [1749728656.024635041] [18259964]: no camera found with serial: 18259964 on try # 4
[camera_driver_node-9] [WARN] [1749728657.025717244] [18259964]: no camera found with serial: 18259964 on try # 5
[camera_driver_node-9] [ERROR] [1749728658.025847924] [18259964]: giving up, camera 18259964 not found!
[camera_driver_node-9] [ERROR] [1749728658.025948209] [flir_camera]: startup failed!
[camera_driver_node-9] [INFO] [1749728658.026012392] [flir_camera]: camera_driver_node started up!
[camera_driver_node-9] [INFO] [1749728658.026457216] [18259964]: rate [Hz] in   0.00 out   0.00 drop   0%
[camera_driver_node-9] [INFO] [1749728661.220521316] [18259964]: rate [Hz] in   0.00 out   0.00 drop   0%
[camera_driver_node-9] [INFO] [1749728666.220537085] [18259964]: rate [Hz] in   0.00 out   0.00 drop   0%
[camera_driver_node-9] [INFO] [1749728671.220483554] [18259964]: rate [Hz] in   0.00 out   0.00 drop   0%

if I manually remove it afterwards:
then it doesnt detect that I have removed it
Any ideas?

[berndpfrommer]

I never tried that, so no surprise it doesn't work. What camera do you use (model)? Particularly interesting if you use USB3 vs GigE.

[GreatAlexander]

Hi @berndpfrommer , for reference we have the same issue (USB3, Model CM3-U3-50S5C).
Our cameras are on a mobile platform (car) and vibrations sometimes cause the cameras to temporarily disconnect, making the ROS driver crash. We have taped the USB3 connectors with conductive tape to minimise this issue.

[berndpfrommer]

I have plans to work on it. I had hoped to postpone this until I implement lifecycle nodes but I can work on it sooner.
Next week is busy but the following one looks better.

[jclinton830]

@berndpfrommer have you made any progress on this front? We are also facing a similar problem. Sometimes, the camera composable node fails, and we need a method to self-heal and bring the composable node back up. Happy to help with the development.

[berndpfrommer]

Sorry folks, I was working on other stuff. But I just managed to get PR #253 over the finish line, so I should be able to start working on this on Wednesday.

[berndpfrommer]

Started to work on lifecycle node support for rolling on this branch. In this branch I will also address robustness to link loss. The branch is in poor shape yet, so please don't use it.