[Question] Lift IK-Rel teleop with mug asset: RigidBodyAPI/root view failures, random object pose on reset, and unstable/jittery teleop control

[Manu752]

I’m using the standard teleoperation script:

./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py
--task Isaac-Lift-*-Franka-IK-Rel-v0 --num_envs 1 --teleop_device keyboard

I modified the Franka Lift IK-Rel config to swap the default cube object to an Isaac mug (SM_Mug_C1). The environment launches correctly (robot + table + mug load fine), but teleoperation becomes very hard to use compared to the cube version:

Motion is jittery/unsmooth and feels “random” (overshoots / oscillates).

The SE(3) keyboard mappings (W/S/A/D/Q/E) don’t feel consistent with the printed axes; it feels like the command frame changes as the robot moves.

This makes it difficult to pick the mug reliably even with num_envs=1.

I did not change the teleop script logic—only the object asset in the Lift env config. The same teleop workflow feels much more stable with the default cube task.

Questions:

Is IK-Rel teleop expected to behave this way (commands interpreted in an EE-local/rotating frame), and is there a recommended setting to make teleop world-aligned / more intuitive?

Are there recommended physics/collision/material settings for non-cube objects (like mugs) to avoid jitter and improve grasp stability during teleop?

IsaacLab 2.3.0
IsaacSim 5.1

[RandomOakForest]

Thank you for posting this. The team will review this shortly and get back to you. @rwiltz for vis. Thank you.

[RandomOakForest]

I'll move this issue to our Discussions for follow up. Apologies for the delay, the team is increasingly tied up with upcoming GTC. Here is a summary to consider, though still under review.

For IK-Rel teleop: yes, the default behavior is effectively end-effector–local (relative) and can feel like the command frame “moves” as the EE moves/rotates. You can make it more world-aligned and less jittery by changing which frame the SE(3) delta is applied in, using smaller gains, and/or switching to the absolute IK task.

1. IK-Rel teleop frame behavior and making it intuitive

In the IK-Rel tasks, the SE(3) teleop agent produces a 6‑D delta pose command (3 translation, 3 rotation).¹
By default, that delta is applied in the task’s EE control frame, so the meaning of “+x/+y/+z” is in the EE/IK frame, not a fixed world frame. This is why the W/S/A/D/Q/E axes feel like they “rotate” with the robot.²¹

To make teleop more intuitive / world-like:

1. Try the absolute IK task variant
   Use the absolute version and see if it feels closer to what you want:

./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py \
  --task Isaac-Lift-Cube-Franka-IK-Abs-v0 --num_envs 1 --teleop_device keyboard

The Abs tasks maintain an absolute pose target and tend to feel more stable for “go here in the world” style control, at the cost of less direct delta control.²

2. Check / modify the SE(3) teleop device config
   The SE(3) controller for keyboard is Se3Keyboard, which maps keys as: reset, gripper toggle, and 3 axes translation + 3 axes rotation.
   Internally, the teleop device returns a 6‑D delta that the task interprets; if you want world‑aligned commands, you can:

   • Look in the task’s action processing / IK controller for where the delta is applied to the EE pose.
   • Change that logic so the translational component is applied in world frame instead of the EE frame (e.g., transform the delta by the world‑to-EE rotation or drop rotation coupling).
     For example, conceptually:
   • Current: $p_{ee}^{new} = p_{ee} \oplus \Delta p_{ee}$ (delta in EE frame).
   • Desired: $p_{ee}^{new} = p_{ee} + R_{world} \Delta x$ where $\Delta x$ is expressed in world coordinates.
     The exact function names can differ, but the key change is “interpret Se3Keyboard linear deltas in world frame before feeding them into IK.”

3. Reduce step size and filter commands
   Jitter in teleop often comes from:

   • Large per‑step deltas from the keyboard.
   • No smoothing of the EE pose command.
     Practical tweaks:
   • Reduce linear and angular step size in the Se3Keyboard config (look for velocity/scale parameters in the teleop device or the SE(3) controller).¹
   • Add a small low‑pass filter on the target EE pose before sending it to the IK solver (e.g., exponential smoothing in the task’s control step).

4. Consider a smoother device for data collection
   NVIDIA documentation explicitly recommends a SpaceMouse for smoother SE(3) teleop and demonstration collection, since it provides continuous analog input instead of discrete key jumps.³¹
   You can try:

./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py \
  --task Isaac-Lift-Cube-Franka-IK-Rel-v0 --num_envs 1 --teleop_device spacemouse

This alone often eliminates much of the oscillatory feel when doing relative IK teleop.³¹

2. Physics/collision/material settings for mug-like objects

Switching from a cube to a thin, curved mug introduces more complex contacts, which can amplify jitter and make grasps unstable if the physics or collision setup isn’t tuned. This is not unique to IK-Rel, but it becomes very noticeable under teleop.

Recommended checks/tweaks for non-cube objects like SM_Mug_C1:

1. Collider quality and shape
   • Avoid overly detailed triangle-mesh colliders for the mug; use a simple convex or a small set of convex parts for the grasp region when possible. Complex meshes increase contact noise.
   • Ensure the mug’s collision approximation roughly matches its outer surface where fingers contact (no thin self‑intersections or tiny features in the collider). NVIDIA forums show that overly complex or thin colliders can produce jittery grasp behavior.⁴
2. Rigid body and contact parameters
   In the rigid body properties and physics scene settings, verify / tune:
   • Mass: Set a realistic but not extremely tiny mass for the mug; very light objects tend to jitter more when squeezed.⁴
   • Max depenetration velocity: Moderate values help avoid explosive separation when small interpenetrations occur. Too large can cause popping/jitter; too small can cause interpenetration.⁴
   • Contact offset / rest offset: Keep them small and consistent across the gripper fingers and the mug, so contacts are not formed or broken at large gaps.⁴
   • Solver iterations / substeps: If grasps jitter, consider increasing the position/velocity iterations for the physics solver or increasing simulation substeps slightly for the manipulation environment.
3. Friction and material setup
   • Use a single, consistent material for the mug and gripper finger pads with a moderate static/dynamic friction (e.g., not extremely low or high).
   • If the mug slides or jitter-bounces between fingers, slightly increasing friction and damping in the contact pair can improve stability.⁴
4. Gripper actuation behavior
   For stable grasps, the fingers should stop closing when the contact forces are sufficient instead of “pushing through” the object indefinitely. NVIDIA examples for grippers suggest:
   • Using position or effort control with compliance (stiffness/damping) on the finger joints so they don’t fight the physics solver.
   • Avoid very high velocity targets or very high stiffness with low damping, which can excite contact jitter.⁵
     Even though you’re using the stock Franka Lift config, changing the object shape may expose aggressive finger gains that worked fine on a cube but cause chatter on a thin mug.
5. Scene scale and gravity consistency
   • Ensure the mug asset’s scale is correct (no accidental huge scaling or tiny scale) and that its center of mass is reasonable (not far outside the shape).
   • Inconsistent scaling plus default solver tolerances can aggravate jitter.

3. Practical steps specific to your setup

Given your description (“only changed the asset”):

1. Quick checks

   • Run the same teleop setup with Isaac-Lift-Cube-Franka-IK-Abs-v0. If teleop feels more intuitive/stable, your issue is mainly frame interpretation (relative vs absolute).²
   • Try --teleop_device spacemouse if you have one, to rule out keyboard step quantization as the main cause of overshoots.³¹

2. For IK-Rel with mug

   • Locate the IK controller / action processing for Isaac-Lift-*-Franka-IK-Rel-v0 and:
     • Reduce the linear/rotational scale factors used to convert Se3Keyboard deltas into pose changes.
     • If you want world-aligned motion, modify the pose update so W/S/A/D/Q/E are applied in world frame, not EE frame.
   • In the mug’s USD or its RigidObjectCfg:
     • Confirm a simple collider (convex, few faces), reasonable mass, and consistent contact offsets.
     • Use a friction material comparable to the cube’s, and avoid extreme friction values.

3. If jitter persists during grasp

   • Increase physics solver iteration counts for the task’s simulation config (position/velocity iterations) and possibly add one more substep for manipulation tasks.
   • Slightly reduce gripper finger stiffness or velocity limits so they “settle” on the mug instead of oscillating.

These changes usually make IK-Rel teleop feel much closer to the cube task, even with a more complex asset like SM_Mug_C1, while keeping the keyboard mappings predictable and the grasp physically stable.

Footnotes

1. https://isaac-sim.github.io/IsaacLab/v2.0.1/source/overview/teleop_imitation.html ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

2. https://isaac-sim.github.io/IsaacLab/main/source/overview/environments.html ↩ ↩² ↩³

3. https://isaac-sim.github.io/IsaacLab/main/source/overview/imitation-learning/teleop_imitation.html ↩ ↩² ↩³

4. https://forums.developer.nvidia.com/t/how-to-improve-stability-when-grasping-an-object/221959 ↩ ↩² ↩³ ↩⁴ ↩⁵

5. https://forums.developer.nvidia.com/t/closing-gripper-around-an-object/325967 ↩