feat: redesign rai_perception API with tiered structure and improve 3D gripping point detection

.github/workflows/poetry-test.yml

@@ -74,7 +74,7 @@ jobs:
         shell: bash
         run: |
           source setup_shell.sh
-          pytest -m "not billable" --cov=./src/rai_core --cov-report=xml --junitxml=junit.xml -o junit_family=legacy
+          pytest -m "not billable and not manual" --cov=./src/rai_core --cov-report=xml --junitxml=junit.xml -o junit_family=legacy
 
       - name: Upload coverage to Codecov
         if: ${{ matrix.ros_distro == 'jazzy' }}

docs/API_documentation/connectors/ROS_2_Connectors.md

@@ -130,6 +130,29 @@ agent = ReActAgent(
 agent.subscribe_source("/from_human", hri_connector)
 ```
 
+## ROS2 Utilities
+
+RAI provides utilities for error handling and parameter access when working with ROS2 connectors:
+
+-   `ROS2ServiceError`: Exception raised for service-related errors (e.g., service unavailable, timeout). Includes service name, timeout duration, and diagnostic suggestions.
+
+-   `ROS2ParameterError`: Exception raised for parameter-related errors (e.g., missing parameter, type mismatch). Includes parameter name, expected type/value, and suggestions for resolution.
+
+-   `get_param_value()`: Utility function for simplified ROS2 parameter access with automatic type conversion and default value support. Reduces boilerplate compared to direct ROS2 parameter API usage.
+
+```python
+from rai.communication.ros2 import ROS2ServiceError, ROS2ParameterError, get_param_value
+
+# Access parameters with defaults
+service_name = get_param_value(node, "/my_service/name", default="/default_service")
+
+# Handle service errors
+try:
+    response = connector.service_call(...)
+except ROS2ServiceError as e:
+    print(f"Service {e.service_name} unavailable: {e.suggestion}")
+```
+
 ## See Also
 
 -   [Connectors Overview](./overview.md)

docs/demos/manipulation.md

@@ -65,16 +65,24 @@ manipulation techniques.
     ros2 launch examples/manipulation-demo.launch.py game_launcher:=demo_assets/manipulation/RAIManipulationDemo/RAIManipulationDemo.GameLauncher
     ```
 
+    By default, perception services register both new model-agnostic service names (`/detection`, `/segmentation`) and legacy names (`/grounding_dino_classify`, `/grounded_sam_segment`) for backward compatibility. To disable legacy names, add the launch argument:
+
+    ```shell
+    ros2 launch examples/manipulation-demo.launch.py game_launcher:=... enable_legacy_service_names:=false
+    ```
+
     2. Run cmd app
 
     ```shell
     python examples/manipulation-demo.py
     ```
 
+    Note: `manipulation-demo.py` uses the new service names (`/detection`, `/segmentation`). Legacy examples like `manipulation-demo-v1.py` use legacy service names and require `enable_legacy_service_names:=true` (default).
+
 2. Interact with the robot arm using natural language commands. For example:
 
     ```
-    "Place every apple on top of the cube"
+    "Place each apple on top of a cube"
     "Build a tower from cubes"
     "Arrange objects in a line"
     "Put two boxes closer to each other. Move only one box."
@@ -134,7 +142,7 @@ manipulation techniques.
 
 The manipulation demo utilizes several components:
 
-1. Vision processing using Grounded SAM 2 and Grounding DINO for object detection and segmentation.
+1. Vision processing using Grounded SAM 2 and Grounding DINO for object detection and segmentation via ROS2 services (`/detection`, `/segmentation`).
 2. RAI agent to process the request and plan the manipulation sequence.
 3. Robot arm control for executing the planned movements.
 
@@ -144,6 +152,21 @@ The main logic of the demo is implemented in the `create_agent` function, which
 examples/manipulation-demo.py
 ```
 
+### Service Names and Backward Compatibility
+
+Perception services support both new model-agnostic service names and legacy names for backward compatibility:
+
+-   New service names (default): `/detection`, `/segmentation`
+-   Legacy service names: `/grounding_dino_classify`, `/grounded_sam_segment`
+
+By default, both sets of names are registered. To disable legacy names (for new applications only), launch with:
+
+```shell
+ros2 launch examples/manipulation-demo.launch.py game_launcher:=... enable_legacy_service_names:=false
+```
+
+Existing applications using legacy service names will continue to work with the default configuration.
+
 ## Known Limitations
 
 -   `Grounding DINO` can't distinguish colors.

examples/embodiments/manipulation_embodiment.json

@@ -3,6 +3,7 @@
         "Always follow the specified payload limit; do not exceed 3 kg.",
         "Before starting the task, by looking at the image, determine if the task can be completed. If yes - proceed without asking. Otherwise report.",
         "Cubes are 5cm tall. When stacking, add 5cm to the z coordinate for each additional cube.",
+        "When placing objects 'on top' of other objects (different object types), you must add a height offset to the target object's z-coordinate. The height offset should be approximately 5cm (0.05m) for most objects. Never use the target object's centroid z-coordinate directly - always add the height offset. For stacking tasks (same object type), follow the stacking rule instead.",
         "Remember to not put objects on top of other objects, unless specifically instructed otherwise",
         "Try to consider the typical size of objects when placing them (width, height, length)",
         "Always leave at least a 5cm margin (x and y) between objects unless specifically instructed otherwise",

examples/manipulation-demo-streamlit.py

@@ -12,6 +12,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import logging
+import os
 from pathlib import Path
 
 import streamlit as st
@@ -36,6 +38,8 @@
 )
 from rai_sim.simulation_bridge import SceneConfig
 
+logger = logging.getLogger(__name__)
+
 
 def launch_description():
     launch_moveit = IncludeLaunchDescription(
@@ -62,6 +66,9 @@ def launch_description():
                 "/launch/openset.launch.py",
             ]
         ),
+        launch_arguments={
+            "enable_legacy_service_names": "true",  # Enable both legacy and new service names for v1/v2 compatibility
+        }.items(),
     )
 
     return LaunchDescription(
@@ -74,16 +81,32 @@ def launch_description():
 
 
 @st.cache_resource
-def initialize_graph():
-    agent, camera_tool = create_agent()
+def initialize_graph(version: str):
+    agent, camera_tool = create_agent(version=version)
     return agent, camera_tool
 
 
 @st.cache_resource
-def initialize_o3de(scenario_path: str, o3de_config_path: str):
+def initialize_o3de(
+    scenario_path: str, o3de_config_path: str, agent_version: str = "v2"
+):
     simulation_config = O3DExROS2SimulationConfig.load_config(
         config_path=Path(o3de_config_path)
     )
+
+    # Adjust required services based on agent version
+    # v1 uses legacy service names, v2 uses new service names
+    if agent_version == "v2":
+        # Replace legacy service names with new ones for v2
+        services = simulation_config.required_robotic_ros2_interfaces["services"]
+        services = [
+            "/detection" if s == "/grounding_dino_classify" else s for s in services
+        ]
+        services = [
+            "/segmentation" if s == "/grounded_sam_segment" else s for s in services
+        ]
+        simulation_config.required_robotic_ros2_interfaces["services"] = services
+
     scene_config = SceneConfig.load_base_config(Path(scenario_path))
     scenario = Scenario(
         task=None,
@@ -143,6 +166,9 @@ def main(o3de_config_path: str):
     # Layout selection in sidebar
     st.sidebar.header("Configuration")
 
+    # Agent version from environment variable
+    agent_version = os.getenv("AGENT_VERSION", "v2")
+
     # Get available scenarios for layout selection
     levels = ["medium", "hard", "very_hard"]
     scenarios: list[Scenario] = get_scenarios(levels=levels)
@@ -260,19 +286,31 @@ def main(o3de_config_path: str):
         else:
             st.sidebar.warning("Demo not initialized yet")
 
+    # Clear conversation history button - resets cached tool results
+    # This forces the agent to query object positions fresh for each new task
+    # instead of using cached ToolMessage results from previous interactions
+    if st.sidebar.button(
+        "Clear Conversation History", help="Clear all previous tool calls and messages"
+    ):
+        st.session_state["messages"] = [
+            AIMessage(content="Hi! I am a robotic arm. What can I do for you?")
+        ]
+        st.rerun()
+
     if "o3de" not in st.session_state:
         selected_scenario_path = get_scenario_path(scenarios, scenario)
         if selected_scenario_path is None:
             st.error(f"Could not find scenario: {scenario}")
             st.stop()
         o3de, initial_scenario = initialize_o3de(
-            selected_scenario_path, o3de_config_path
+            selected_scenario_path, o3de_config_path, agent_version=agent_version
         )
         st.session_state["o3de"] = o3de
         st.session_state["current_scenario"] = initial_scenario
 
     if "graph" not in st.session_state:
-        graph, camera_tool = initialize_graph()
+        logger.info(f"Initializing agent version: {agent_version}")
+        graph, camera_tool = initialize_graph(version=agent_version)
         st.session_state["graph"] = graph
         st.session_state["camera_tool"] = camera_tool
 

examples/manipulation-demo-v1.py

@@ -0,0 +1,37 @@
+# Copyright (C) 2025 Robotec.AI
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import logging
+from typing import List
+
+from langchain_core.messages import BaseMessage, HumanMessage
+from manipulation_common import create_agent
+
+logger = logging.getLogger(__name__)
+
+
+def main():
+    agent, _ = create_agent(version="v1")
+    messages: List[BaseMessage] = []
+
+    while True:
+        prompt = input("Enter a prompt: ")
+        messages.append(HumanMessage(content=prompt))
+        output = agent.invoke({"messages": messages})
+        output["messages"][-1].pretty_print()
+
+
+if __name__ == "__main__":
+    main()

examples/manipulation-demo.launch.py

@@ -32,6 +32,13 @@ def generate_launch_description():
         description="Path to the game launcher executable",
     )
 
+    # Declare argument for legacy service names (forwarded to openset.launch.py)
+    enable_legacy_arg = DeclareLaunchArgument(
+        "enable_legacy_service_names",
+        default_value="true",
+        description="Enable legacy service names for backward compatibility (default: true)",
+    )
+
     launch_game_launcher = ExecuteProcess(
         cmd=[
             LaunchConfiguration("game_launcher"),
@@ -64,11 +71,17 @@ def generate_launch_description():
                 "/launch/openset.launch.py",
             ]
         ),
+        launch_arguments={
+            "enable_legacy_service_names": LaunchConfiguration(
+                "enable_legacy_service_names"
+            ),
+        }.items(),
     )
 
     return LaunchDescription(
         [
             game_launcher_arg,
+            enable_legacy_arg,
             launch_game_launcher,
             launch_openset,
             launch_moveit,

examples/manipulation-demo.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2024 Robotec.AI
+# Copyright (C) 2025 Robotec.AI
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,23 +14,53 @@
 
 
 import logging
+import signal
 from typing import List
 
+import rclpy
 from langchain_core.messages import BaseMessage, HumanMessage
-from manipulation_common import create_agent
 
 logger = logging.getLogger(__name__)
 
 
+def create_agent():
+    """Create and configure the manipulation agent (v2).
+
+    This is a thin wrapper around manipulation_common.create_agent(version="v2").
+    See manipulation_common.py for documentation on parameter configuration.
+    """
+    from manipulation_common import create_agent as _create_agent
+
+    agent, _ = _create_agent(version="v2")
+    return agent
+
+
 def main():
-    agent, _ = create_agent()
+    agent = create_agent()
     messages: List[BaseMessage] = []
 
-    while True:
-        prompt = input("Enter a prompt: ")
-        messages.append(HumanMessage(content=prompt))
-        output = agent.invoke({"messages": messages})
-        output["messages"][-1].pretty_print()
+    def cleanup(signum, frame):
+        """Cleanup handler for graceful shutdown."""
+        logger.info("Shutting down...")
+        if rclpy.ok():
+            rclpy.shutdown()
+        exit(0)
+
+    signal.signal(signal.SIGINT, cleanup)
+
+    try:
+        while True:
+            prompt = input("Enter a prompt: ")
+            messages.append(HumanMessage(content=prompt))
+            output = agent.invoke({"messages": messages})
+            output["messages"][-1].pretty_print()
+    except KeyboardInterrupt:
+        cleanup(None, None)
+    except Exception as e:
+        logger.error(f"Error: {e}")
+        if rclpy.ok():
+            rclpy.shutdown()
+        raise
 
 
 if __name__ == "__main__":