[Question] Best workflow for attaching Deformable Gripper to Articulation for RL? (Sim 5.1.0 / Lab 2.3.0)

[conner9403]

[Question] Best workflow for attaching Deformable Gripper to Articulation for RL? (Sim 5.1.0 / Lab 2.3.0)

Build Info

• Isaac Lab Version: 2.3.0 (Planning to migrate)
• Isaac Sim Version: 5.1.0 (pip installed)
• OS: Windows 11
• GPU: NVIDIA GeForce RTX 5090 Laptop GPU (Driver: 591.44)

Context & Goal

I am setting up a reinforcement learning environment in Isaac Lab for topology optimization of a soft gripper (Fin Ray effect). My goal is to attach a Deformable Body (gripper finger) to a Franka Emika Panda robot (Articulation) and measure deformation during grasping.

Research on Known Issues:
I am aware that interactions between Deformable Bodies and Articulations have historically been a known limitation. I found a recent discussion (Jan 2025) stating that "deformable bodies cannot be part of an articulation" (NVIDIA Forum Link).
However, it is unclear if this restriction still applies to Attachment APIs in version 5.1.0, or if there is a specific workaround supported by Isaac Lab.

Current Behavior

When I attempt to attach the Deformable Body to the Franka's end-effector link (panda_hand) in Isaac Sim 5.1.0:

1. The deformable mesh does not follow the robot arm during motion (Attachment ignored).
2. The mesh falls through the floor or floats at the spawn position.
3. In some configurations, the simulation becomes unstable or freezes.

What I Tried

I have exhaustively tested multiple approaches to resolve this:

1. PhysxPhysicsAttachment (Direct): Deformable -> Articulation Link.
   • Result: Attachment ignored; mesh falls.
2. Dummy Rigid Body + FixedJoint: Created an intermediate anchor Rigid Body, connected it to the robot hand via FixedJoint, and then attached the Deformable to this anchor.
   • Result: The robot arm became physically locked (immobile). (Suspecting collision issues between the dummy body and the hand).
3. Kinematic/Dynamic Anchors + Physics Callback: Tried syncing an anchor body's position via physics step callbacks.
   • Result: Attachment still fails or lags.
4. USD Hierarchy Parenting: Placed the Deformable under the robot's link hierarchy.
   • Result: [Error] missing xformstack reset. Simulation freezes or behaves unpredictably.

My Questions

1. Status Confirmation: Based on the forum discussion, is direct attachment between Deformable Body and Articulation Link officially unsupported in Isaac Sim 5.1.0?
2. Best Practice: If direct attachment is impossible, what is the recommended "Best Practice" workaround for this setup in Isaac Lab?
3. Dummy Body Workaround: Is the "Dummy Rigid Body" approach the correct way to go? (My attempt caused the robot to lock up—should I disable collisions for the dummy body?)
4. Low-Level Access: Can I access PhysX's low-level kinematic soft body features (e.g., per-vertex inverse mass, GPU kinematic target buffer) via Isaac Lab APIs to manually force the mesh to follow the gripper?

Any advice on establishing a stable link for RL training would be greatly appreciated.

[RandomOakForest]

Thank you for posting these questions and the details of what you have tried. We will aim to address this post shortly. Thank you for your interest in Isaac Lab.

[RandomOakForest]

Apologies for the late reply. I will move this post to our Discussions section for follow up. Here is a summary to consider with key things to try that may be of help.

Direct attachment of a deformable gripper to a Franka articulation is still effectively unsupported for your use‑case in 5.1.0 / Isaac Lab 2.3.0, and the best practice today is to (1) keep the soft body passive and (2) drive it via kinematic control or rigid “actuator” bodies that follow the robot, not by making the soft body part of the articulation itself. Below is a practical workflow that fits your RL setup.¹²³⁴

---

1. Status: deformable ↔ articulation attachment

• Isaac Lab maintainers currently treat deformables as passive objects; there is no supported, documented workflow to make a deformable body a true articulation link in Isaac Lab 2.x.³
• PhysX itself allows rigid–deformable attachments in USD/Omniverse, but integrating that into Isaac Lab’s articulation handling is not guaranteed to work and is explicitly called out as not expected to work “currently” in Lab.³
• So for your purpose (Fin‑Ray finger attached to Franka hand, controlled via RL), “deformable as articulation link” should be considered unsupported in 5.1.0 / Lab 2.3.0.

---

2. Best‑practice workflow in Isaac Lab

For RL with a soft gripper, a robust pattern is:

1. Spawn the Franka as a standard articulation (Isaac Lab Articulation API).²
2. Spawn the soft gripper as a separate deformable asset using the Lab deformable tutorial pattern (e.g., DeformableObject asset class).¹
3. Use kinematic control of a subset of deformable nodes to “attach” the gripper base to the robot hand:
   • Identify nodes (or a region) at the base of the finger that should follow panda_hand.
   • Every physics step: compute the gripper base pose from the Franka end‑effector pose, transform those nodes to this pose, and write them as kinematic targets.
   • Leave the remaining nodes free so FEM can produce Fin‑Ray‑style deformation under contact.
4. Use Isaac Lab’s regular observation buffers to read nodal positions / derived deformation metrics for RL.¹

This keeps the robot–soft coupling purely through kinematics and contact forces, which is consistent with how the deformable tutorials are set up.⁵¹

High‑level pseudo‑loop (Python) in Isaac Lab style:

# each physics step
franka_ee_pose = franka.get_link_pose("panda_hand")          # articulated link pose
node_positions = deformable.get_nodal_positions()            # (N, 3) or (N, 4)

# pick a subset of nodes belonging to the base
base_indices = base_node_indices  # precomputed once

# build kinematic target buffer
kin_targets = deformable.get_zero_kinematic_target_buffer()  # shape (N, 4) or similar

for idx in base_indices:
    local_pos = base_local_positions[idx]  # position relative to base frame
    world_pos = franka_ee_pose.transform(local_pos)
    kin_targets[idx, :3] = world_pos
    kin_targets[idx, 3] = 0.0  # inverse mass or flag depending on API

deformable.write_nodal_kinematic_target_to_sim(kin_targets)

The exact API names depend on the current Lab version, but the core operations—getting nodal positions, creating a kinematic target buffer, and writing it to the sim—are documented in the deformable tutorial.¹

---

3. Dummy rigid body approach: when and how

Using a dummy rigid body is conceptually valid, but you need two key safeguards:

1. Disable collisions between:
   • Dummy body and Franka links it is fixed to (to avoid self‑collisions that lock the arm).
   • Dummy body and parts of the deformable that should not be “crushed” by the anchor.
2. Use the dummy as a kinematic driver, not as a heavy dynamic body:
   • Set the dummy to kinematic and update its pose from panda_hand every step; or
   • Use a high‑stiffness joint but keep mass/inertia small and collision filters correct.

In practice, this pattern is most useful if you want to:

• Attach multiple rigid “studs” into the deformable, then use joints (distance, prismatic, etc.) between these studs to actuate the soft body, as suggested in PhysX/Omniverse docs.⁴³
• Keep the FEM object passive but drive it via a “skeleton” of rigid drivers.

However, given you already saw the arm lock, I would recommend:

• If you keep the dummy‑body route:
  • Mark the dummy as kinematic; drive it from the Franka pose.
  • Set collision filters so it does not collide with Franka links.
  • Attach the deformable to the dummy via Omniverse attachment (so FEM sees the rigid anchor) and tune soft parameters to avoid extreme forces.
• If you just want a soft finger following the hand for RL, skip the dummy body and use direct nodal kinematic control (section 2), which is simpler and more stable for training.

---

4. Low‑level PhysX soft‑body control from Isaac Lab

PhysX offers fine‑grained control over soft bodies:

• Vertices can be made partially kinematic by setting inverse mass to zero and providing a kinematic target buffer.⁴
• Kinematic targets are stored in a GPU buffer and updated each step; PhysX computes consistent velocities, which is important for realistic contact.⁴

Isaac Lab exposes this in a high‑level way rather than direct PxSoftBody pointers:

• The deformable tutorial shows how to construct a per‑node kinematic target array and write it into the simulation via a method like write_nodal_kinematic_target_to_sim on the deformable asset wrapper.¹
• This gives you per‑vertex control similar in spirit to the raw PhysX API (inverse mass flags + targets) without directly managing the GPU buffer yourself.⁴¹

For RL and topology optimization, this is usually sufficient:

• You can:
  • Select which nodes are kinematic (base anchored to Franka) vs free (rest of finger).
  • Modify which nodes are constrained as part of the “topology” or design variable.
  • Read back nodal positions/velocities to compute deformation, strain proxies, or task rewards.¹

If you truly need raw PhysX objects (e.g., to bypass Lab abstractions), that typically requires writing a custom Omniverse extension or native plugin; this is not part of the standard Isaac Lab Python API and will fight the RL stack, so I would avoid it unless absolutely necessary.⁵⁴

---

Recommended concrete path for your project

1. Treat the Franka and soft finger as separate assets.
2. Use per‑node kinematic targets to attach the soft finger base to panda_hand (no articulation link membership).
3. If you still want rigid anchors, create kinematic dummy rigid bodies, fully disable self‑collisions with the robot, and attach them to the deformable, but only after you have the pure kinematic‑node approach working.
4. Use Isaac Lab deformable APIs for node control instead of trying to access PhysX’s GPU soft‑body buffers directly; they are designed for exactly this style of control and integrate with the RL loop.⁵¹

Footnotes

1. https://isaac-sim.github.io/IsaacLab/main/source/tutorials/01_assets/run_deformable_object.html ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹

2. https://isaac-sim.github.io/IsaacLab/main/source/tutorials/01_assets/run_articulation.html ↩ ↩²

3. https://github.com/isaac-sim/IsaacLab/discussions/1327 ↩ ↩² ↩³ ↩⁴

4. https://nvidia-omniverse.github.io/PhysX/physx/5.4.1/docs/SoftBodies.html ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

5. https://isaac-sim.github.io/IsaacLab/main/source/how-to/simulation_performance.html ↩ ↩² ↩³