Skip to content

Add Remote Rerun Viewer #2243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+22 −6
Open

Add Remote Rerun Viewer #2243

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion src/lerobot/scripts/lerobot_record.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,10 @@ class RecordConfig:
policy: PreTrainedConfig | None = None
# Display all cameras on screen
display_data: bool = False
# Display data on a remote Rerun server
display_url: str = None
Copy link

Copilot AI Oct 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] Align the type of display_url with the project's union style by using str | None for consistency with other fields (e.g., teleop_time_s: float | None).

Suggested change
display_url: str = None
display_url: str | None = None

Copilot uses AI. Check for mistakes.
# Port of the remote Rerun server
display_port: int = 9876
# Use vocal synthesis to read events.
play_sounds: bool = True
# Resume recording on an existing dataset.
Expand Down Expand Up @@ -374,7 +378,7 @@ def record(cfg: RecordConfig) -> LeRobotDataset:
init_logging()
logging.info(pformat(asdict(cfg)))
if cfg.display_data:
init_rerun(session_name="recording")
init_rerun(session_name="recording", url=cfg.display_url, port=cfg.display_port)

robot = make_robot_from_config(cfg.robot)
teleop = make_teleoperator_from_config(cfg.teleop) if cfg.teleop is not None else None
Expand Down
6 changes: 5 additions & 1 deletion src/lerobot/scripts/lerobot_teleoperate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,10 @@ class TeleoperateConfig:
teleop_time_s: float | None = None
# Display all cameras on screen
display_data: bool = False
# Display data on a remote Rerun server
display_url: str = None
Copy link

Copilot AI Oct 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] Type annotation for display_url is inconsistent with the rest of the config (which uses the PEP 604 style, e.g., float | None). For consistency, use str | None for display_url.

Suggested change
display_url: str = None
display_url: str | None = None

Copilot uses AI. Check for mistakes.
# Port of the remote Rerun server
display_port: int = 9876


def teleop_loop(
Expand Down Expand Up @@ -186,7 +190,7 @@ def teleoperate(cfg: TeleoperateConfig):
init_logging()
logging.info(pformat(asdict(cfg)))
if cfg.display_data:
init_rerun(session_name="teleoperation")
init_rerun(session_name="teleoperation", url=cfg.display_url, port=cfg.display_port)

teleop = make_teleoperator_from_config(cfg.teleop)
robot = make_robot_from_config(cfg.robot)
Expand Down
16 changes: 12 additions & 4 deletions src/lerobot/utils/visualization_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,19 +16,23 @@
import os
from typing import Any

import cv2
Copy link

Copilot AI Oct 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] Importing OpenCV at module import time forces a heavy optional dependency even when visualization is disabled. Move the import inside log_rerun_data (or behind the JPEG branch) to avoid unnecessary runtime/import errors on setups without OpenCV.

Copilot uses AI. Check for mistakes.
import numpy as np
import rerun as rr

from .constants import OBS_PREFIX, OBS_STR


def init_rerun(session_name: str = "lerobot_control_loop") -> None:
def init_rerun(session_name: str = "lerobot_control_loop", url: str = None, port: int = 9876) -> None:
"""Initializes the Rerun SDK for visualizing the control loop."""
batch_size = os.getenv("RERUN_FLUSH_NUM_BYTES", "8000")
os.environ["RERUN_FLUSH_NUM_BYTES"] = batch_size
rr.init(session_name)
memory_limit = os.getenv("LEROBOT_RERUN_MEMORY_LIMIT", "10%")
rr.spawn(memory_limit=memory_limit)
if url:
rr.connect_grpc(url=f"rerun+http://{url}:{port}/proxy")
Copy link

Copilot AI Oct 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Python SDK exposes rr.connect, not rr.connect_grpc; this will raise an AttributeError. Use rr.connect with host:port (e.g., rr.connect(f"{url}:{port}")) rather than a rerun+http URL with /proxy.

Suggested change
rr.connect_grpc(url=f"rerun+http://{url}:{port}/proxy")
rr.connect(f"{url}:{port}")

Copilot uses AI. Check for mistakes.
else:
rr.spawn(memory_limit=memory_limit)


def _is_scalar(x):
Expand All @@ -48,7 +52,7 @@ def log_rerun_data(
to the Rerun viewer. It handles different data types appropriately:
- Scalars values (floats, ints) are logged as `rr.Scalars`.
- 3D NumPy arrays that resemble images (e.g., with 1, 3, or 4 channels first) are transposed
from CHW to HWC format and logged as `rr.Image`.
from CHW to HWC format, encoded as JPEG and logged as `rr.EncodedImage`.
- 1D NumPy arrays are logged as a series of individual scalars, with each element indexed.
- Other multi-dimensional arrays are flattened and logged as individual scalars.

Expand All @@ -75,7 +79,11 @@ def log_rerun_data(
for i, vi in enumerate(arr):
rr.log(f"{key}_{i}", rr.Scalars(float(vi)))
else:
rr.log(key, rr.Image(arr), static=True)
_, buffer = cv2.imencode(
".jpg", cv2.cvtColor(arr, cv2.COLOR_RGB2BGR), [int(cv2.IMWRITE_JPEG_QUALITY), 50]
Comment on lines +82 to +83
Copy link

Copilot AI Oct 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This path assumes 3-channel RGB input; it will fail or produce incorrect results for 1-channel (grayscale) and 4-channel (RGBA) images mentioned in the docstring. Handle channel count explicitly: no color conversion for single-channel, RGB→BGR for 3-channel, and RGBA→BGR (drop alpha) for 4-channel before JPEG encoding.

Suggested change
_, buffer = cv2.imencode(
".jpg", cv2.cvtColor(arr, cv2.COLOR_RGB2BGR), [int(cv2.IMWRITE_JPEG_QUALITY), 50]
# Handle channel count explicitly for JPEG encoding
if arr.ndim == 3:
if arr.shape[2] == 1:
# Grayscale, no color conversion needed
arr_to_encode = arr
elif arr.shape[2] == 3:
# RGB to BGR
arr_to_encode = cv2.cvtColor(arr, cv2.COLOR_RGB2BGR)
elif arr.shape[2] == 4:
# RGBA to BGR (drop alpha)
arr_to_encode = cv2.cvtColor(arr[:, :, :3], cv2.COLOR_RGB2BGR)
else:
raise ValueError(f"Unsupported number of channels for image encoding: {arr.shape[2]}")
else:
raise ValueError(f"Expected 3D array for image encoding, got shape {arr.shape}")
_, buffer = cv2.imencode(
".jpg", arr_to_encode, [int(cv2.IMWRITE_JPEG_QUALITY), 50]

Copilot uses AI. Check for mistakes.
)
Comment on lines +82 to +84
Copy link

Copilot AI Oct 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] cv2.imencode expects uint8 data; if arr is float (e.g., [0,1]) or another dtype, encoding may fail or yield incorrect output. Consider converting/scaling to uint8 prior to encoding and checking the boolean return value of cv2.imencode to handle failures.

Copilot uses AI. Check for mistakes.
encoded_image = buffer.tobytes()
rr.log(key, rr.EncodedImage(contents=encoded_image, media_type="image/jpeg"), static=True)
Copy link

Copilot AI Oct 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logged type should be rr.ImageEncoded, not rr.EncodedImage; rr.EncodedImage does not exist in the Python API and will fail at runtime. Replace with rr.ImageEncoded(contents=..., media_type="image/jpeg").

Suggested change
rr.log(key, rr.EncodedImage(contents=encoded_image, media_type="image/jpeg"), static=True)
rr.log(key, rr.ImageEncoded(contents=encoded_image, media_type="image/jpeg"), static=True)

Copilot uses AI. Check for mistakes.

if action:
for k, v in action.items():
Expand Down