ACT+ALOHA, can only get 66% success

[raul-machine-learning]

Ticket Type

🐛 Bug Report (Something isn't working)

Environment & System Info

$ uv run lerobot-info
- LeRobot version: 0.6.1
- Platform: Linux-6.8.0-124-generic-x86_64-with-glibc2.35
- Python version: 3.12.12
- Huggingface Hub version: 1.22.0
- Transformers version: 5.5.4
- Datasets version: 4.8.5
- Numpy version: 2.2.6
- FFmpeg version: 4.4.2-0ubuntu0.22.04.1
- PyTorch version: 2.11.0+cu128
- Torchcodec version: 0.11.1+cpu
- Is PyTorch built with CUDA support?: True
- Cuda version: 12.8
- GPU model: NVIDIA A100 80GB PCIe
- Using GPU in script?: <fill in>
- lerobot scripts: ['lerobot-annotate', 'lerobot-calibrate', 'lerobot-dataset-viz', 'lerobot-edit-dataset', 'lerobot-eval', 'lerobot-find-cameras', 'lerobot-find-joint-limits', 'lerobot-find-port', 'lerobot-imgtransform-viz', 'lerobot-info', 'lerobot-record', 'lerobot-replay', 'lerobot-rollout', 'lerobot-setup-can', 'lerobot-setup-motors', 'lerobot-teleoperate', 'lerobot-train', 'lerobot-train-tokenizer']

Description

I believe this configuration is essentially the "Hello World" example for LeRobot, and I'd like to get it working reliably before moving on to Pi0.5. Unfortunately, I found the documentation and execution steps somewhat unclear, and with the settings I've tried so far, I'm seeing a relatively low success rate.

Context & Reproduction

My LeRobot version is based on commit: 73dbb6f

Training script

#!/usr/bin/env bash
set -e

export MUJOCO_GL=egl

# Train ACT on ALOHA transfer cube task
# Saves checkpoints every 1000 steps for evaluation

uv run lerobot-train \
  --policy.type=act \
  --dataset.repo_id=lerobot/aloha_sim_transfer_cube_human \
  --output_dir=./outputs/aloha_transfer_cube_act \
  --job_name=aloha_transfer_act \
  --batch_size=8 \
  --steps=100000 \
  --env_eval_freq=10000 \
  --eval.n_episodes=50 \
  --save_checkpoint=true \
  --save_freq=10000 \
  --policy.device=cuda \
  --policy.push_to_hub=false \
  --wandb.enable=true \
  --wandb.project=lerobot_aloha

Evaluation script

#!/usr/bin/env bash
set -e

unset CONDA_DEFAULT_ENV CONDA_PREFIX VIRTUAL_ENV
export CUDA_VISIBLE_DEVICES=1
export MUJOCO_GL=egl

LATEST=$(ls -d ./outputs/aloha_transfer_cube_act/checkpoints/*/ 2>/dev/null | sort -V | tail -n 1)
LATEST=${LATEST%/}/pretrained_model

echo "Evaluating: $LATEST"

uv run lerobot-eval \
  --policy.path="$LATEST" \
  --env.type=aloha \
  --env.task=AlohaTransferCube-v0 \
  --eval.n_episodes=50 \
  --eval.batch_size=1 \
  --policy.device=cuda

Relevant logs or stack trace

Checklist

• I have searched existing tickets to ensure this isn't a duplicate.
• I am using the latest version of the main branch.
• I have verified this is not an environment-specific problem.

Additional Info / Workarounds

I'd be great if you could provide sample scripts so we can train/evaluate the models