LeTeleop: Isaac Teleop Backend for LeRobot

[jiwenc-nv]

LeTeleop: Isaac Teleop Backend for LeRobot

Overview

This document describes the integration plan for using NVIDIA Isaac Teleop as a teleoperation backend within the LeRobot framework. The goal is to implement new LeRobot Teleoperator subclasses that wrap Isaac Teleop's TeleopSession, exposing its diverse input devices — XR (VR controllers, hand tracking, full-body tracking), peripheral (foot pedals, Manus gloves), and custom sources — as first-class teleop backends in LeRobot's recording and control pipelines.

Motivation

LeRobot's existing teleop backends are predominantly hardware-specific leader arms (SO100, Koch), phones, and gamepads. Isaac Teleop brings:

• Diverse device support:
  • XR devices (VR controllers, hand tracking, head pose, full-body tracking) via OpenXR
  • non-XR peripherals (foot pedals, Manus gloves, joysticks) via the plugin system
• Retargeting engine: A graph-based pipeline for transforming raw input data from any source into robot-specific commands (SE3 poses, dexterous hand joints, gripper commands, locomotion velocities)
• Extensible device abstraction: Plugin architecture allows adding new input devices — both XR and non-XR — without modifying core code
• Existing LeRobot bridge: A minimal examples/lerobot/record.py in the Teleop repo already demonstrates recording tracking data to LeRobot dataset format (head + wrist positions only)

This integration makes Isaac Teleop's full input pipeline — XR and non-XR alike — available through LeRobot's standard Teleoperator interface, enabling demonstration recording, leader-follower control, and policy training with a wide range of input devices.

Architecture

How the Two Systems Map

Isaac Teleop Concept	LeRobot Concept	Notes
TeleopSession	Internal state of Teleoperator	Wraps the session lifecycle (OpenXR + plugins)
TeleopSession.step() → RetargeterIO	Teleoperator.get_action() → RobotAction	Main data extraction point
TeleopSessionConfig + pipeline graph	TeleoperatorConfig dataclass	Declarative configuration
BaseRetargeter pipeline	ProcessorStep pipeline (optional)	Retargeting can happen on either side
OptionalTensorGroup	dict[str, float | np.ndarray]	Data format conversion
Session lifecycle (OpenXR + plugins)	connect() / disconnect()	Resource management

Data Flow

block
    columns 1

    block:lerobot["LeRobot"]
        columns 5

        F["Robot.send_action()"]
        space:1
        E["Record loop / control pipeline<br/>ProcessorStep (EE → joints IK)"]
        space:1
        D["LeTeleop Teleoperator<br/>.get_action() → RobotAction"]
    end

    space:1

    block:nvidia["NVIDIA Isaac Teleop"]
        columns 5
        A["Device Interface<br/>(VR headset, controllers,<br/>hand tracking, manus gloves)"]
        space:1
        B["Retargeting Pipeline<br/>(Se3, DexHand, Gripper,<br/>Locomotion)"]
        space:1
        C["TeleopSession.step() →<br/>Dict[str, OptionalTensorGroup]"]
    end

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F

    style A fill:#76b900,color:#fff
    style B fill:#76b900,color:#fff
    style C fill:#76b900,color:#fff
    style D fill:#ff9d00,color:#fff
    style E fill:#ff9d00,color:#fff
    style F fill:#ff9d00,color:#fff
    style nvidia fill:none,stroke:#76b900,stroke-width:2px
    style lerobot fill:none,stroke:#ff9d00,stroke-width:2px

Loading

Components of Isaac Teleop

For LeRobot developers unfamiliar with Isaac Teleop, here is a quick tour of its layered architecture. Think of it as a pipeline: hardware sources produce raw tracking data, retargeters transform that data into robot-meaningful commands, and the session orchestrates the whole loop.

Sources (hardware → tensors)

Sources are stateless converters that poll hardware trackers and produce OptionalTensorGroup outputs. The built-in sources use OpenXR, but the architecture is not XR-only — plugins provide additional source types (e.g., foot pedals via Linux joystick, Manus gloves via Manus SDK). Each source auto-registers its tracker; you don't manage hardware directly.

Source	What it captures	Output shape
HandsSource	26 hand joints per hand (OpenXR hand tracking or Manus gloves)	Joint positions (26,3), orientations (26,4)
ControllersSource	Controller aim/grip pose + buttons, triggers, thumbsticks	Position (3,), orientation (4,), trigger/squeeze/thumbstick scalars
HeadSource	Head pose (HMD or other tracker)	Position (3,), orientation (4,)
FullBodySource	Full-body skeleton (Pico format)	Joint positions (N,3), orientations (N,4), validity flags
Generic3AxisPedalSource	Foot pedal (left, right, rudder) via Linux joystick	3 normalized scalars [-1, 1]

All outputs are OptionalTensorGroup — they have an .is_none property that is True when the input is unavailable (hand occluded, controller off, pedal disconnected, etc.). This is the Isaac Teleop equivalent of returning None.

Retargeters (tensors → robot commands)

Retargeters are pure-Python transform nodes that implement BaseRetargeter. Each declares its input_spec() and output_spec(), then implements _compute_fn(). They are wired together via .connect() to form a DAG.

Retargeter	Input	Output	Use case
Se3AbsRetargeter	Hand or controller pose	EE position + quaternion	Arm end-effector control
Se3RelRetargeter	Hand or controller pose	Delta EE pose	Relative/incremental arm control
GripperRetargeter	Hand joint positions	Scalar 0.0–1.0	Gripper open/close from pinch distance
DexHandRetargeter	26 hand joints	Robot hand joint angles	Dexterous hand control (requires URDF + YAML config)
LocomotionRootCmdRetargeter	Controller thumbstick	Linear/angular velocity + height	Mobile base locomotion
TensorReorderer	Multiple tensor groups	Single flat array	Packing multiple outputs into one action vector

LeRobot analogy: Retargeters are similar to LeRobot's ProcessorStep — both are chainable transforms. The difference is that Isaac Teleop retargeters run inside the teleop session (before get_action()), while LeRobot processors run after (between get_action() and send_action()).

OutputCombiner (merging outputs)

OutputCombiner takes named outputs from multiple sources and retargeters and merges them into a single dict. This is the "finalization" step that defines what TeleopSession.step() returns.

pipeline = OutputCombiner({
    "ee_pose": se3_retargeter.output("ee_pose"),
    "gripper": gripper_retargeter.output("gripper_command"),
    "locomotion": locomotion_retargeter.output("root_command"),
})

TeleopSession (the orchestrator)

TeleopSession is the top-level context manager that ties everything together. On construction, it auto-discovers which sources and trackers the pipeline needs. Each call to .step():

1. Polls all hardware trackers (OpenXR and plugin-provided)
2. Feeds raw data through the retargeting pipeline
3. Returns the final Dict[str, OptionalTensorGroup]

LeRobot analogy: TeleopSession is conceptually the "inner loop" that our _IsaacTeleopBase.get_action() wraps. LeRobot developers never interact with TeleopSession directly — the IsaacTeleopController / IsaacTeleopHand subclasses handle it.

Plugins (optional hardware extensions)

Isaac Teleop supports plugins for devices beyond the built-in OpenXR sources:

Plugin	Purpose
Manus hand plugin	Manus glove hand tracking via Manus SDK
OAK camera	OAK camera video capture with H.264 encoding
3-axis pedal	Linux joystick foot pedal for locomotion input
Synthetic hands	Generates fake hand tracking from controller poses (useful for testing)

Where Retargeting Happens

Isaac Teleop has its own retargeting engine (raw tracking → robot commands). LeRobot has a ProcessorStep pipeline (action transforms, IK/FK). Two strategies:

1. Isaac-side retargeting (recommended for dexterous hands, locomotion): Configure Isaac Teleop's pipeline with appropriate retargeters (DexHandRetargeter, Se3AbsRetargeter, GripperRetargeter, etc.) so that TeleopSession.step() already outputs robot-ready joint positions. The LeRobot teleoperator simply maps these to RobotAction keys.

2. LeRobot-side retargeting (recommended for simple EE control of arms): Pass through raw SE3 poses from Isaac Teleop, then use LeRobot's existing EEReferenceAndDelta + InverseKinematicsEEToJoints processor steps to convert to joint targets for a specific follower robot.

Both approaches can coexist; the config determines which retargeters are in the Isaac pipeline.

Scope

In Scope (Phase 1: Core Integration)

• Shared base class _IsaacTeleopBase: session lifecycle, plugin management, connect/disconnect
• IsaacTeleopController ("isaac_teleop_controller"): VR controller aim pose + trigger/squeeze as gripper
• IsaacTeleopHand ("isaac_teleop_hand"): Wrist pose + finger joint positions from hand tracking (OpenXR or Manus gloves)
• Configuration per type: Separate TeleoperatorConfig subclass per semantic type
• Pipeline builders: ControllersSource / HandsSource → optional retargeters → OutputCombiner
• Data format bridge: Conversion from OptionalTensorGroup tensors to LeRobot RobotAction dicts

In Scope (Phase 2: Advanced Features)

• IsaacTeleopFullBody ("isaac_teleop_full_body"): Body joint positions for humanoid teleoperation
• BiIsaacTeleopController ("bi_isaac_teleop_controller"): Bimanual dual-controller setup (composition pattern)
• BiIsaacTeleopHand ("bi_isaac_teleop_hand"): Bimanual hand tracking
• Retargeter configs: DexHandRetargeter, Se3AbsRetargeter, GripperRetargeter, TriHandMotionControllerRetargeter as config-level options within existing subclasses
• Locomotion commands: Joystick/pedal-based root velocity commands for mobile robots
• Feedback channel: send_feedback() for haptic feedback via controllers (if supported by the device)
• Camera streaming: OAK camera plugin integration for robot-eye-view in VR

Out of Scope

• Modifying Isaac Teleop's core C++ layer or OpenXR runtime
• ROS2 integration (Isaac Teleop already has a ROS2 example; this integration is Python-native)
• Training/inference pipeline changes in LeRobot (the teleoperator is purely an input device)
• CloudXR networking setup (deployment concern, not integration concern)

Teleoperator Subclassing Strategy

The Design Tension

Isaac Teleop's TeleopSession is fundamentally config-driven: one class handles all modes. The pipeline graph (sources + retargeters + OutputCombiner) determines both what hardware is active and what data flows out. Isaac Lab follows this pattern exactly — one IsaacTeleopDevice class, with a pipeline_builder callable in config that returns an OutputCombiner. Different robots and control modes are entirely config-level variations.

LeRobot's convention is different. Studying existing Teleoperator subclasses reveals a clear pattern for when subclasses are created:

Reason to subclass	Example	Why not config?
Different communication protocol	IOSPhone vs AndroidPhone (HEBI SDK vs WebXR)	Different threading model, SDK, connection logic
Different semantic output space	KeyboardTeleop vs KeyboardEndEffectorTeleop vs KeyboardRoverTeleop	action_features returns fundamentally different keys and types
Compositional relationship	BiSOLeader wraps two SOLeader instances	Different lifecycle, prefixes keys with left_/right_

And when config alone suffices:

Reason for config only	Example
Feature toggle (gripper on/off)	GamepadTeleopConfig.use_gripper
Numeric parameters (speed, thresholds)	KeyboardRoverTeleopConfig.linear_speed
Hardware variant, identical behavior	SO100Leader = SO101Leader = SOLeader (aliases)
Modular part selection	Reachy2TeleoperatorConfig.with_l_arm, .with_r_arm, .with_neck

Recommended Approach: Subclass by Semantic Output

Isaac Teleop modes produce fundamentally different output semantics — a controller returns SE3 pose + trigger, hand tracking returns 26 joint poses, full-body returns a skeleton, a foot pedal returns velocity scalars. These map to different action_features shapes. Following LeRobot convention, each semantic output type warrants its own subclass.

However, within a semantic type, pipeline configuration (which retargeters to use, which robot's URDF) is handled by config — not subclassing.

classDiagram
    class _IsaacTeleopBase {
        <<abstract>>
        session lifecycle
        connect / disconnect
        plugin management
        +_build_pipeline()* OutputCombiner
    }

    class IsaacTeleopController {
        +action_features
        ee_pos : ndarray shape 3
        ee_quat : ndarray shape 4
        gripper : float
        ---
        Config: hand_side,
        gripper_source, retargeter (se3)
    }

    class IsaacTeleopHand {
        +action_features
        wrist_pos : ndarray shape 3
        wrist_quat : ndarray shape 4
        joint_positions : ndarray shape 26x3
        ---
        Config: hand_side,
        retargeter (dex), retargeter_config
    }

    class IsaacTeleopFullBody {
        +action_features
        joint_positions : ndarray shape Nx3
        joint_orientations : ndarray shape Nx4
        joint_valid : ndarray shape N
        ---
        Config: (none yet)
    }

    _IsaacTeleopBase <|-- IsaacTeleopController
    _IsaacTeleopBase <|-- IsaacTeleopHand
    _IsaacTeleopBase <|-- IsaacTeleopFullBody

Loading

Bimanual variants follow LeRobot's composition pattern (like BiSOLeader):

class BiIsaacTeleopController(Teleoperator):
    """Wraps two IsaacTeleopController instances (left + right)."""

Why Not a Single Class with a mode String?

A single IsaacTeleop class with mode: str = "controller" would work but conflicts with LeRobot conventions:

1. action_features would be mode-dependent — a consumer can't know the output shape without also knowing the mode. In LeRobot, action_features is a static contract per class.
2. Config registration — LeRobot uses @TeleoperatorConfig.register_subclass("type_name") for CLI dispatch (--teleop.type=isaac_teleop_controller). Separate types are more discoverable.
3. Precedent — Keyboard has three subclasses for three output types, not one class with a mode enum.

Why Not One Subclass Per Pipeline?

Going the other direction — one subclass per pipeline preset (e.g., IsaacTeleopDexHand, IsaacTeleopSe3Controller) — would be too granular. The retargeting choice changes the resolution of the output, not its semantic type. A controller with Se3AbsRetargeter still outputs {ee_pos, ee_quat, gripper} — the retargeter just transforms the coordinates. This matches the Reachy2 pattern: one class with config-driven internal variation.

Detailed Design

Package Structure

lerobot/teleoperators/isaac_teleop/
    __init__.py              # Exports all classes and configs
    _base.py                 # _IsaacTeleopBase — shared session lifecycle
    controller.py            # IsaacTeleopController(Teleoperator)
    hand.py                  # IsaacTeleopHand(Teleoperator)
    full_body.py             # IsaacTeleopFullBody(Teleoperator)
    bimanual.py              # BiIsaacTeleopController, BiIsaacTeleopHand
    config.py                # All config dataclasses
    pipelines.py             # Pipeline builder functions (sources + retargeters → OutputCombiner)
    tensor_conversion.py     # OptionalTensorGroup → RobotAction conversion utilities

Configuration

Each semantic type gets its own config, registered separately for CLI discoverability:

@dataclass
class _IsaacTeleopBaseConfig(TeleoperatorConfig):
    """Shared config fields for all Isaac Teleop-based teleoperators."""
    app_name: str = "LeTeleop"
    plugins: list[str] | None = None

@TeleoperatorConfig.register_subclass("isaac_teleop_controller")
@dataclass
class IsaacTeleopControllerConfig(_IsaacTeleopBaseConfig):
    hand_side: str = "right"                   # "left" or "right"
    gripper_source: str = "trigger"            # "trigger" or "squeeze"
    retargeter: str | None = None              # "se3_abs", "se3_rel", or None (raw)
    retargeter_config_path: str | None = None  # Path to retargeter YAML/JSON

@TeleoperatorConfig.register_subclass("isaac_teleop_hand")
@dataclass
class IsaacTeleopHandConfig(_IsaacTeleopBaseConfig):
    hand_side: str = "right"
    retargeter: str | None = None              # "dex_hand" or None (raw)
    retargeter_config_path: str | None = None
    hand_urdf: str | None = None               # Required for dex_hand retargeter

@TeleoperatorConfig.register_subclass("isaac_teleop_full_body")
@dataclass
class IsaacTeleopFullBodyConfig(_IsaacTeleopBaseConfig):
    pass  # No additional config needed yet

@TeleoperatorConfig.register_subclass("bi_isaac_teleop_controller")
@dataclass
class BiIsaacTeleopControllerConfig(_IsaacTeleopBaseConfig):
    gripper_source: str = "trigger"
    retargeter: str | None = None
    retargeter_config_path: str | None = None

Core Implementation

Shared base (not registered, not directly instantiable):

class _IsaacTeleopBase(Teleoperator):
    """Shared lifecycle for all Isaac Teleop-based teleoperators."""

    def __init__(self, config: _IsaacTeleopBaseConfig):
        super().__init__(config)
        self.config = config
        self._session: TeleopSession | None = None
        self._connected = False

    @property
    def is_connected(self) -> bool:
        return self._connected

    @property
    def is_calibrated(self) -> bool:
        return True  # Tracking devices are self-calibrating

    def calibrate(self) -> None:
        pass

    def configure(self) -> None:
        pass

    def connect(self, calibrate: bool = True) -> None:
        pipeline = self._build_pipeline()  # Subclass-specific
        session_config = TeleopSessionConfig(
            app_name=self.config.app_name,
            pipeline=pipeline,
            plugins=[...],
        )
        self._session = TeleopSession(session_config)
        self._session.__enter__()
        self._connected = True

    @abc.abstractmethod
    def _build_pipeline(self) -> OutputCombiner:
        """Subclass builds its specific source + retargeter pipeline."""
        ...

    def send_feedback(self, feedback: dict[str, Any]) -> None:
        pass  # Phase 2

    def disconnect(self) -> None:
        if self._session:
            self._session.__exit__(None, None, None)
            self._session = None
        self._connected = False

Controller teleoperator:

class IsaacTeleopController(_IsaacTeleopBase):
    config_class = IsaacTeleopControllerConfig
    name = "isaac_teleop_controller"

    @property
    def action_features(self) -> dict:
        return {
            "ee_pos": np.ndarray,    # shape (3,)
            "ee_quat": np.ndarray,   # shape (4,)
            "gripper": float,        # 0.0 (open) to 1.0 (closed)
        }

    @property
    def feedback_features(self) -> dict:
        return {}

    def _build_pipeline(self) -> OutputCombiner:
        controllers = ControllersSource(name="controllers")
        # Optionally chain retargeters (Se3AbsRetargeter, etc.)
        return build_controller_pipeline(controllers, self.config)

    def get_action(self) -> RobotAction:
        result = self._session.step()
        side = self.config.hand_side
        controller = result[f"controller_{side}"]

        if controller.is_none:
            raise TrackingLostError(f"{side} controller not tracked")

        return {
            "ee_pos": np.asarray(controller[ControllerInputIndex.AIM_POSITION]),
            "ee_quat": np.asarray(controller[ControllerInputIndex.AIM_ORIENTATION]),
            "gripper": _read_gripper(controller, self.config.gripper_source),
        }

Bimanual controller (composition pattern matching BiSOLeader):

class BiIsaacTeleopController(_IsaacTeleopBase):
    config_class = BiIsaacTeleopControllerConfig
    name = "bi_isaac_teleop_controller"

    @property
    def action_features(self) -> dict:
        return {
            "left_ee_pos": np.ndarray, "left_ee_quat": np.ndarray, "left_gripper": float,
            "right_ee_pos": np.ndarray, "right_ee_quat": np.ndarray, "right_gripper": float,
        }

    def _build_pipeline(self) -> OutputCombiner:
        controllers = ControllersSource(name="controllers")
        return build_bimanual_controller_pipeline(controllers, self.config)

    def get_action(self) -> RobotAction:
        result = self._session.step()
        action = {}
        for side in ("left", "right"):
            controller = result[f"controller_{side}"]
            if controller.is_none:
                raise TrackingLostError(f"{side} controller not tracked")
            action[f"{side}_ee_pos"] = np.asarray(controller[ControllerInputIndex.AIM_POSITION])
            action[f"{side}_ee_quat"] = np.asarray(controller[ControllerInputIndex.AIM_ORIENTATION])
            action[f"{side}_gripper"] = _read_gripper(controller, self.config.gripper_source)
        return action

Tensor-to-Action Conversion

The core bridge logic converts Isaac Teleop's OptionalTensorGroup output to LeRobot's flat RobotAction dict:

def convert_controller_to_action(result: RetargeterIO, config: IsaacTeleopConfig) -> RobotAction:
    side = config.hand_side
    controller = result[f"controller_{side}"]

    if controller.is_none:
        raise TrackingLostError(f"{side} controller not tracked")

    action = {}
    action["ee_pos"] = np.asarray(controller[ControllerInputIndex.AIM_POSITION])    # (3,)
    action["ee_quat"] = np.asarray(controller[ControllerInputIndex.AIM_ORIENTATION]) # (4,)

    # Gripper from trigger or squeeze
    if config.gripper_source == "trigger":
        raw = float(controller[ControllerInputIndex.TRIGGER_VALUE])
    else:
        raw = float(controller[ControllerInputIndex.SQUEEZE_VALUE])
    action["gripper"] = raw

    return action

For retargeted outputs (e.g., dexterous hand joints), the conversion reads the retargeter's named output tensors and maps them to joint-named keys matching the target robot.

Pipeline Builders

Each subclass's _build_pipeline() creates an Isaac Teleop pipeline graph internally. The pipelines.py module provides builder functions:

Teleoperator Class	Sources	Optional Retargeters	action_features
IsaacTeleopController	ControllersSource	Se3AbsRetargeter, Se3RelRetargeter	ee_pos (3,), ee_quat (4,), gripper
IsaacTeleopHand	HandsSource	DexHandRetargeter	wrist_pos (3,), wrist_quat (4,), joint_positions (26,3)
IsaacTeleopFullBody	FullBodySource	None	joint_positions (N,3), joint_orientations (N,4), joint_valid (N,)
BiIsaacTeleopController	ControllersSource (L+R)	Se3AbsRetargeter	left_ee_pos, left_ee_quat, left_gripper, right_*
BiIsaacTeleopHand	HandsSource (L+R)	DexHandRetargeter	left_wrist_pos, left_joint_positions, right_*

Retargeters change the coordinate frame or resolution of the output, not its semantic type. A controller with Se3AbsRetargeter still outputs {ee_pos, ee_quat, gripper} — the retargeter maps poses from XR-space to robot-space.

Usage Examples

Recording demonstrations with VR controller:

python -m lerobot.scripts.lerobot_record \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyUSB0 \
    --teleop.type=isaac_teleop_controller \
    --teleop.hand_side=right \
    --teleop.gripper_source=trigger

Bimanual hand tracking with dexterous retargeting:

python -m lerobot.scripts.lerobot_record \
    --robot.type=custom_bimanual \
    --teleop.type=bi_isaac_teleop_hand \
    --teleop.retargeter=dex_hand \
    --teleop.retargeter_config_path=./config/my_hand.yaml

Simple teleoperation script:

from lerobot.teleoperators.isaac_teleop import IsaacTeleopController, IsaacTeleopControllerConfig
from lerobot.robots.so_follower import SO100Follower, SO100FollowerConfig

teleop_config = IsaacTeleopControllerConfig(hand_side="right")
robot_config = SO100FollowerConfig(port="/dev/ttyUSB0")

with IsaacTeleopController(teleop_config) as teleop, SO100Follower(robot_config) as robot:
    while True:
        action = teleop.get_action()  # {"ee_pos": ..., "ee_quat": ..., "gripper": ...}
        # User must set up a processor pipeline for EE → joint conversion
        robot.send_action(processed_action)

Dependencies

Python Version Compatibility

Project	Supported Python Versions
LeRobot	>= 3.12 (classifiers: 3.12, 3.13)
Isaac Teleop	3.10, 3.11, 3.12
Overlap	3.12 only

This is a significant constraint. The only compatible Python version today is 3.12.

• Isaac Teleop does not support Python 3.13+ (LeRobot's secondary supported version). Isaac Teleop uses pybind11 C++ extensions that must be compiled per Python minor version, so adding 3.13 support requires a build/test pass on that version.
• LeRobot does not support Python 3.10 or 3.11 — it uses 3.12+ syntax features (e.g., type statement, T | None in non-annotation contexts) and pins requires-python = ">=3.12".

Action required: Pin the integration to Python 3.12. Longer term, either extend Isaac Teleop's build matrix to include 3.13, or accept 3.12 as the single supported version for this integration.

Required

• isaac_teleop Python package (built from /code/Teleop/ via CMake → wheel)
• OpenXR runtime (SteamVR, Monado, or CloudXR) — required for XR sources; non-XR plugins work without it
• numpy >= 2.0
• Python 3.12 (the only version supported by both projects)

Optional

• scipy — for Se3RelRetargeter
• dex_retargeting — for DexHandRetargeter
• torch — if using dex_retargeting with optimization

Build Requirement

Isaac Teleop must be built and its Python wheel installed:

cd /code/Teleop
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build
pip install build/python_package/Release/isaac_teleop-*.whl

Key Risks and Mitigations

Risk	Impact	Mitigation
OpenXR runtime availability	XR-based sources require an OpenXR runtime	Support headless/mock mode via Isaac Teleop's synthetic hands plugin for testing
Latency from retargeting pipeline	Delayed control response	Profile; most retargeters are pure numpy/scipy, sub-ms. Dex retargeting with optimization is heavier — benchmark and document limits
C++ build dependency	Complex build for users	Provide pre-built wheels; document minimal build steps; consider conda package
Tracking loss	OptionalTensorGroup.is_none during occlusion	Return last-known action with a tracking_valid flag; let downstream decide hold/stop behavior
API version coupling	Isaac Teleop API changes break integration	Pin to a specific Isaac Teleop version; wrap low-level access in tensor_conversion.py for easy adaptation

Testing Strategy

1. Unit tests with synthetic hands: Isaac Teleop's synthetic hands plugin provides deterministic test input without XR hardware. Write tests that verify get_action() returns correct shapes and values.
2. Mock teleoperator: Extend LeRobot's existing MockTeleop pattern to simulate Isaac Teleop outputs.
3. Integration test: End-to-end recording test using synthetic hands → IsaacTeleop.get_action() → LeRobotDataset.add_frame().
4. Hardware-in-the-loop: Manual testing with actual devices — VR headset, foot pedals, Manus gloves (documented in test plan, not CI).

Implementation Plan

Phase 1: Core Integration (MVP)

• Create package skeleton: lerobot/teleoperators/isaac_teleop/ with _base.py, controller.py, hand.py, config.py, pipelines.py, tensor_conversion.py
• Implement _IsaacTeleopBase: Shared session lifecycle, connect/disconnect, plugin management
• Implement IsaacTeleopController: VR controller → {ee_pos, ee_quat, gripper} output
• Implement IsaacTeleopHand: Hand tracking → {wrist_pos, wrist_quat, joint_positions} output
• Register in factory: Add "isaac_teleop_controller" and "isaac_teleop_hand" to make_teleoperator_from_config()
• Write unit tests: Using synthetic hands plugin
• Verify recording: End-to-end test with lerobot_record script
• Python 3.13: Add full support for Python 3.13 in Isaac Teleop

Phase 2: Advanced Modes

• BiIsaacTeleopController: Composition of left + right controllers (mirrors BiSOLeader pattern)
• BiIsaacTeleopHand: Composition of left + right hand tracking
• IsaacTeleopFullBody: Body joint output for humanoid teleoperation
• Retargeter configs: Wire up DexHandRetargeter, Se3AbsRetargeter, GripperRetargeter as config-level options
• Locomotion: Joystick/pedal → velocity commands (may warrant its own subclass)

Phase 3: Polish & Scale

• Extra embodiments: How to host more sim assets for embodiments (e.g. Sharpa hands, AgiBot, etc.)
• Feedback channel: Haptic feedback via send_feedback()
• Camera plugin: OAK camera streaming as LeRobot camera source
• Documentation: User guide with setup instructions for supported devices (VR headsets, foot pedals, Manus gloves)
• Pre-built wheels: CI pipeline for Isaac Teleop wheel distribution

References

• Isaac Teleop architecture: docs/source/overview/architecture.rst
• Isaac Teleop LeRobot example: examples/lerobot/record.py
• Isaac Teleop ROS2 publisher (reference for data extraction patterns): examples/teleop_ros2/python/teleop_ros2_publisher.py
• Isaac Teleop OutputCombiner (pipeline graph composition)
• Isaac Teleop TensorReorderer (tensor flattening/reordering for custom robots)
• Isaac Teleop index constants: ControllerInputIndex, HandInputIndex, FullBodyInputIndex
• LeRobot teleoperator base class: Teleoperator
• LeRobot phone teleoperator (non-hardware reference): teleop_phone.py
• LeRobot recording script: lerobot_record.py