Suction Grasping Package

A robust framework designed to estimate 6DoF suction grasp poses in camera coordinate systems using RGB images and scene geometry. The geometry can be obtained via stereo matching (using the CREStereo backend) or directly from external depth/point cloud inputs.

🔬 Pipeline

The pipeline processes input images and geometric data to identify the optimal grasp pose.

1. Object Segmentation: Detects target objects using RF-DETR.
2. Geometry Generation: Computes depth via CREStereo (from a stereo pair) or parses external depth/point cloud data.
3. Suitability Mapping: Estimates surface normals and flatness to build a suitability heatmap.
4. 3D Evaluation: Voxelizes the region of interest and applies orientation-aware 3D convolutions.
5. Pose Selection: Determines the best 6DoF grasp pose in the camera frame.

For details on the internal implementation details and modules, see the Internal Documentation.

🔌 API & Integration

The primary external integration is handled by the RobotPickerAdapter class (located in grasp_pose/adapters/robot_picker.py). It implements a contract compatible with the robot-picker-template.

demo.mp4

Adapter Interface

from grasp_pose import RobotPickerAdapter

# Initialize the adapter
adapter = RobotPickerAdapter(config_path="configs/pipeline_config_template.yaml")

# (Optional) Update stereo images if using CREStereo depth backend
adapter.update_stereo_images(left_ir, right_ir, baseline_m=None)

# Predict the optimal grasp
position, rpy, bbox_xyxy, confidence, grasp_mode = adapter.predict_grasps(
    image=image_bgr,
    depth=depth_map,  # Pass None if using the internal CREStereo backend
    intrinsics_params=[fx, fy, cx, cy],
    roi_xywh=(x, y, w, h)
)

Return Values

Return Item	Type	Description
position	np.ndarray(3,)	Grasp position $(X, Y, Z)$ in meters in camera coordinates.
rpy	np.ndarray(3,)	Grasp orientation in Euler Roll-Pitch-Yaw angles (radians).
bbox_xyxy	np.ndarray(4,)	2D bounding box $[x_{min}, y_{min}, x_{max}, y_{max}]$ in pixels.
confidence	float	Pose prediction confidence score in $[0.0, 1.0]$.
grasp_mode	int	Suction cup configuration ID ($0, 1,$ or $2$).

Geometry Backends

The adapter supports two backend modes, configurable via runtime.depth_backend in configs/pipeline_config_template.yaml:

• crestereo: Builds scene geometry internally from a stereo pair.
  • Call update_stereo_images(left_ir, right_ir, baseline_m) with the latest frames before calling predict_grasps(..., depth=None).
• depth: Uses external geometry.
  • Pass a depth map (HxW uint16 in mm or float in meters) or a point cloud (HxWx3 float) directly to the depth parameter of predict_grasps.

🛠️ Requirements & Installation

Requirements

• OS: Linux
• Python: >=3.9 (Python 3.10 recommended)
• CUDA: Toolkit and PyTorch with compatible CUDA version.
• Weights:
  • RF-DETR (Required): Place in data/rfdetr.pth (or set via models.segmentation.rfdetr.weights).
  • CREStereo (Optional): Required for the stereo backend. Place in data/crestereo.pth.
  • Download weights from the Google Drive.

Installation

# Setup virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Update core package managers
python -m pip install -U pip setuptools wheel

# Install current package in editable mode
python -m pip install -e .

🚀 Running the System

Desktop Client (robot-picker-template)

The client provides interactive visualization and tests real-time grasp calculation. It is a fork of the original client template integrating the RobotPickerAdapter.

Important

Run commands from the robot-picker-template/src directory to ensure relative imports like camera.* resolve correctly.

Run with a pre-recorded .bag file:

cd robot-picker-template/src
python3 -c "from descktop_client.client import Viewer; Viewer('./../data/basket.bag').start()"

Run with a live Intel RealSense camera:

cd robot-picker-template/src
python3 -c "from descktop_client.client import Viewer; Viewer(None).start()"

Run in 3D Point Cloud Debug Mode: Launch the client with the GRASP_POINTCLOUD_DEBUG_VIEW=1 environment variable to visualize the 3D point cloud and grasp space:

cd robot-picker-template/src
GRASP_POINTCLOUD_DEBUG_VIEW=1 python3 -c "from descktop_client.client import Viewer; Viewer('./../data/basket.bag').start()"

Controls

• Ctrl + LMB or MMB: Trigger grasp estimation on the current frame.
• Space: Save the current frame and metadata JSON to robot-picker-template/logs/.
• q: Exit.

Environment Variables

• GRASP_POINTCLOUD_DEBUG_VIEW=1: Enables the 3D debug point cloud viewer.
• GRASP_POINTCLOUD_ROI_PADDING_PX=40: Sets point cloud rendering padding around the ROI.

Offline Profiling

Run the profiling script on a configured bag file:

python test/main.py