feat: redesign rai_perception API with tiered structure and improve 3D gripping point detection #750
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #750 +/- ##
==========================================
+ Coverage 65.79% 69.74% +3.95%
==========================================
Files 78 82 +4
Lines 3403 3514 +111
==========================================
+ Hits 2239 2451 +212
+ Misses 1164 1063 -101 ☔ View full report in Codecov by Sentry. |
Juliaj
changed the title
Juliaj
force-pushed
the
jj/feat/3dpipe_and_usability
branch
from
7aca530 to
09674aa
Compare
Juliaj
force-pushed
the
jj/feat/3dpipe_and_usability
branch
from
32d8802 to
4af6d1b
Compare
|
Regarding rai_sim and rai_bench, I don't think a lot of changes will be required. Mainly naming changes, switching to new tools or changing paths. For example in |
|
I will start with coderabbit to speed things up. @coderabbitai full review |
|
✅ Actions performedFull review triggered. |
WalkthroughMajor architectural refactoring and expansion of the RAI perception module introducing new algorithms, services, and components alongside deprecation of legacy agents. Updates include new ROS2 communication utilities, timeout decorators, comprehensive gripping-point extraction pipeline, model registries, and extensive test coverage. Multiple packages reorganized with service-oriented architecture replacing direct agent instantiation. Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
Note
Due to the large number of review comments, Critical, Major severity comments were prioritized as inline comments.
🤖 Fix all issues with AI agents
In
`@src/rai_extensions/rai_perception/rai_perception/components/detection_publisher.py`:
- Around line 401-411: The image subscription callback image_callback currently
calls _process_image which can raise ROS2ServiceError; catch ROS2ServiceError
inside image_callback (around the call to _process_image) so the exception does
not escape the callback, log a clear message via the node/process logger
including the service name and error details, and return early to skip
processing this image; keep other exceptions unchanged or optionally log them
separately to avoid disrupting the executor.
In
`@src/rai_extensions/rai_perception/rai_perception/components/gripping_points.py`:
- Around line 55-62: Add a configurable timeout to
PointCloudFromSegmentationConfig (e.g., service_timeout: float =
Field(default=15.0, ...)) and update all places that call get_future_result to
pass this config value instead of relying on the 5s default; specifically modify
the code paths in this module that call get_future_result (referenced as
get_future_result) so they accept/obtain the PointCloudFromSegmentationConfig
instance and use its service_timeout when calling get_future_result (also update
any constructors or callers that instantiate PointCloudFromSegmentationConfig to
propagate the new field).
- Around line 629-707: The package is missing the scikit-learn dependency used
by _filter_dbscan, _filter_kmeans_largest, _filter_isolation_forest, and
_filter_lof; add scikit-learn = ">=1.0.0, <1.4.0" to the dependencies section of
rai_perception's pyproject.toml
(src/rai_extensions/rai_perception/pyproject.toml) so runtime imports of
sklearn.cluster, sklearn.ensemble, and sklearn.neighbors do not raise
ImportError.
In
`@src/rai_extensions/rai_perception/rai_perception/components/service_utils.py`:
- Around line 75-94: The function check_service_available currently calls
wait_for_ros2_services which treats timeout 0 as “wait indefinitely”; add a
guard at the top of check_service_available that if timeout_sec <= 0 performs a
non-blocking lookup using the connector’s node service registry (e.g., call
connector.node.get_service_names_and_types() and check whether service_name
appears) and return True/False accordingly, otherwise keep the existing
try/except around wait_for_ros2_services(connector, [service_name],
timeout=timeout_sec) returning True on success and False on
TimeoutError/ValueError.
In `@src/rai_extensions/rai_perception/rai_perception/components/topic_utils.py`:
- Around line 106-145: The detection-only branch currently assigns the method
object to detection_service and never sets segmentation_service, causing the
subsequent runtime check to always raise; fix by treating tools with a
service_name attribute as detection-only: if hasattr(tool, "service_name") set
detection_service = tool.service_name() if callable (or tool.service_name
otherwise), leave segmentation_service as None, and do not break validation that
requires GetObjectGrippingPointsTool for topics — instead adjust the later
validation so required_services = [detection_service] plus segmentation_service
if not None, and only require required_topics/get_config() when a
GetObjectGrippingPointsTool is present; ensure references in your edits point to
GetObjectGrippingPointsTool, service_name (call or attribute),
detection_service, segmentation_service, get_config, and required_topics.
In `@src/rai_semap/rai_semap/ros2/node.py`:
- Line 28: The project imports rai_perception in node.py (from
rai_perception.components.perception_utils import extract_pointcloud_from_bbox)
but pyproject.toml lacks that dependency; open rai_semap/pyproject.toml and add
rai_perception to the [tool.poetry.dependencies] (or equivalent) section—either
as a path-based dependency like the existing rai_core entry or with an
appropriate version constraint—then run your dependency tool (poetry
install/update) to update lockfiles so the package is available at runtime.
🟡 Minor comments (23)
src/rai_extensions/rai_perception/rai_perception/components/perception_utils.py-1-1 (1)
1-1: Typo in copyright header."Robotech.AI" should be "Robotec.AI".
Proposed fix
-# Copyright (C) 2025 Robotech.AI +# Copyright (C) 2025 Robotec.AIsrc/rai_extensions/rai_perception/rai_perception/agents/_helpers.py-30-31 (1)
30-31: Avoid callingPath.home()in default argument.
Path.home()is evaluated once at function definition time, not at each call. This can cause issues in testing or if the home directory changes during runtime.Proposed fix
+_DEFAULT_WEIGHTS_PATH = Path.home() / Path(".cache/rai") + + def create_service_wrapper( service_class: Type, ros2_name: str, model_name: str, service_name: str, - weights_root_path: str | Path = Path.home() / Path(".cache/rai"), + weights_root_path: str | Path | None = None, ) -> tuple[ROS2Connector, object]: """Create a service instance with ROS2 parameters configured. ... """ + if weights_root_path is None: + weights_root_path = _DEFAULT_WEIGHTS_PATH + ros2_connector = ROS2Connector(ros2_name, executor_type="single_threaded")tests/rai_perception/test_mocks.py-21-68 (1)
21-68: Silence unused mock parameters to keep ruff clean.These mock signatures are correct, but ruff’s ARG002 will flag unused arguments. Prefix them with
_to keep the interface while avoiding lint failures.🧹 Suggested cleanup
- def get_boxes(self, image_msg, classes, box_threshold, text_threshold): + def get_boxes(self, _image_msg, classes, _box_threshold, _text_threshold): @@ - def get_segmentation(self, image, boxes): + def get_segmentation(self, _image, _boxes): @@ - def get_boxes(self, image_msg, classes, box_threshold, text_threshold): + def get_boxes(self, _image_msg, _classes, _box_threshold, _text_threshold): @@ - def get_segmentation(self, image, boxes): + def get_segmentation(self, _image, _boxes):tests/rai_perception/conftest.py-88-125 (1)
88-125: Mock node doesn't persist declared parameters.
declare_parameteris aMagicMock, so any code that declares and then reads parameters will raiseParameterNotDeclaredException. The_parametersdict is properly used byget_parameter,has_parameter, andset_parameters, butdeclare_parameterneeds to actually store parameters to keep the mock aligned with ROS2 behavior.🧩 Suggested mock fix
+ def declare_parameter(name, value=None): + from rclpy.parameter import Parameter + param = Parameter(name, value=value) + _parameters[name] = param + return param @@ - mock_node.declare_parameter = MagicMock() + mock_node.declare_parameter = declare_parametersrc/rai_extensions/rai_perception/rai_perception/components/visualization_utils.py-161-185 (1)
161-185: Docstring contradicts implementation: function documentsRaises: RuntimeErrorbut actually catches exceptions and returns original points.The docstring states the function raises
RuntimeErroron transform failure, but the implementation catches all exceptions and returns the original points instead. This mismatch can confuse callers who might try to handle the exception.🐛 Proposed fix: Update docstring to match behavior
Returns: List of transformed 3D points (Nx3 arrays) in target frame. Returns original points if transform fails. - - Raises: - RuntimeError: If transform lookup fails """examples/manipulation-demo-v1.py-1-1 (1)
1-1: Minor: Copyright year inconsistency.This file uses
2024while other new files in this PR use2025. Consider updating for consistency.docs/api_design_considerations.md-241-242 (1)
241-242: Minor grammar issue: hyphenation."Multi Agent Systems" should be hyphenated as "Multi-Agent Systems" for grammatical correctness.
📝 Suggested fix
-7. **Rachwał et al. (2025)** - "RAI: Flexible Agent Framework for Embodied AI" - https://arxiv.org/abs/2505.07532 - Introduces the RAI framework for creating embodied Multi Agent Systems for robotics, with integration for ROS 2, Large Language Models, and simulations. Describes the framework's architecture, tools, and mechanisms for agent embodiment. +7. **Rachwał et al. (2025)** - "RAI: Flexible Agent Framework for Embodied AI" - https://arxiv.org/abs/2505.07532 + Introduces the RAI framework for creating embodied Multi-Agent Systems for robotics, with integration for ROS 2, Large Language Models, and simulations. Describes the framework's architecture, tools, and mechanisms for agent embodiment.docs/api_design_considerations.md-78-78 (1)
78-78: Typographical error: double period.The sentence ends with
mechanisms.".which has a period both inside and outside the quote.📝 Suggested fix
-The [RAI framework paper](https://arxiv.org/abs/2505.07532}) (Rachwał et al., 2025) states Tools are "compatible with langchain, enabling seamless integration with tool-calling-enabled LLMs" while also being "used by Agents utilizing other decision-making mechanisms.". Based on this paper, RAI serves four distinct roles at different architectural levels: +The [RAI framework paper](https://arxiv.org/abs/2505.07532) (Rachwał et al., 2025) states Tools are "compatible with langchain, enabling seamless integration with tool-calling-enabled LLMs" while also being "used by Agents utilizing other decision-making mechanisms." Based on this paper, RAI serves four distinct roles at different architectural levels:Note: Also removes the extra
}in the URL.src/rai_extensions/rai_perception/follow-ups.md-87-88 (1)
87-88: Use heading syntax instead of italic emphasis.markdownlint flags emphasis used as a heading.
✅ Replace emphasis with heading
-_Proposed Solution: Service-Level Abstraction_ +#### Proposed Solution: Service-Level Abstractionsrc/rai_extensions/rai_perception/follow-ups.md-5-12 (1)
5-12: Fix list indentation to satisfy markdownlint.The TOC list uses extra indentation; reduce to standard two-space nesting.
✅ Lint-friendly list indentation
-- [Breaking Changes](`#breaking-changes`) - - [Agent Service Renaming and Service Name Changes](`#agent-service-renaming-and-service-name-changes`) - - [Agents and Vision Markup Deprecation](`#agents-and-vision-markup-deprecation`) - - [Related Package Updates](`#related-package-updates`) -- [Post-PR Follow-ups](`#post-pr-follow-ups`) - - [Generic Detection Tools Abstraction](`#generic-detection-tools-abstraction`) - - [Progressive Evaluation: GetDistanceToObjectsTool](`#progressive-evaluation-getdistancetoobjectstool`) - - [Service Name Updates in rai_bench](`#service-name-updates-in-rai_bench`) +- [Breaking Changes](`#breaking-changes`) + - [Agent Service Renaming and Service Name Changes](`#agent-service-renaming-and-service-name-changes`) + - [Agents and Vision Markup Deprecation](`#agents-and-vision-markup-deprecation`) + - [Related Package Updates](`#related-package-updates`) +- [Post-PR Follow-ups](`#post-pr-follow-ups`) + - [Generic Detection Tools Abstraction](`#generic-detection-tools-abstraction`) + - [Progressive Evaluation: GetDistanceToObjectsTool](`#progressive-evaluation-getdistancetoobjectstool`) + - [Service Name Updates in rai_bench](`#service-name-updates-in-rai_bench`)tests/rai_perception/agents/test_run_perception_agents.py-12-13 (1)
12-13: License header appears truncated.Line 12 reads "See the specific language governing" but should be "See the License for the specific language governing" to match the standard Apache 2.0 license text used in other files.
📝 Suggested fix
-# See the specific language governing permissions and +# See the License for the specific language governing permissions andsrc/rai_extensions/rai_perception/rai_perception/services/base_vision_service.py-36-62 (1)
36-62: Missing validation forWEIGHTS_URLbefore download attempt.The class validates that
WEIGHTS_FILENAMEis set (line 49-50), butWEIGHTS_URLdefaults to an empty string and isn't validated. If a subclass forgets to setWEIGHTS_URLand weights don't exist,download_weightswill be called with an empty URL, causing a confusing wget failure.Proposed fix to validate WEIGHTS_URL
def __init__( self, weights_root_path: str | Path = DEFAULT_WEIGHTS_ROOT_PATH, ros2_name: str = "", ros2_connector: Optional[ROS2Connector] = None, ): # TODO: After agents are deprecated, make ros2_connector a required parameter # (remove Optional and default None). Services should always receive a connector. if not self.WEIGHTS_FILENAME: raise ValueError("WEIGHTS_FILENAME is not set") + if not self.WEIGHTS_URL: + raise ValueError("WEIGHTS_URL is not set") self.weights_root_path = Path(weights_root_path)docs/extensions/rethinking_usability.md-216-216 (1)
216-216: Fix typo: "avaialble" → "available".-Three tools are avaialble for RAI agents:`GetDetectionTool`, `GetObjectGrippingPointsTool` and `GetDistanceToObjectsTool`. +Three tools are available for RAI agents: `GetDetectionTool`, `GetObjectGrippingPointsTool` and `GetDistanceToObjectsTool`.tests/rai_perception/services/test_run_perception_services.py-71-88 (1)
71-88: Silence Ruff ARG001 for unused tracking args.
*args/**kwargsare unused in these helpers, which Ruff flags. Rename to underscore-prefixed to avoid lint failures.✅ Suggested tweak
- def track_connector_init(*args, **kwargs): + def track_connector_init(*_args, **_kwargs): call_order.append("connector_init") return MagicMock() - def track_detection_init(*args, **kwargs): + def track_detection_init(*_args, **_kwargs): call_order.append("detection_init") mock_instance = MagicMock() mock_instance.run.side_effect = lambda: call_order.append("detection_run") return mock_instance - def track_segmentation_init(*args, **kwargs): + def track_segmentation_init(*_args, **_kwargs): call_order.append("segmentation_init") mock_instance = MagicMock() mock_instance.run.side_effect = lambda: call_order.append( "segmentation_run" ) return mock_instancesrc/rai_core/rai/communication/ros2/parameters.py-22-24 (1)
22-24: Resolve or track the TODO before release.If this reconsideration is still open, please capture it in an issue/ADR so it doesn’t get lost.
Do you want me to draft an issue/ADR template for this decision?
src/rai_extensions/rai_perception/rai_perception/vision_markup/segmenter.py-40-55 (1)
40-55: Align docstring with actual config_path handling.The docstring says
config_pathis ignored, but it’s forwarded tosuper().__init__. Please clarify to avoid confusing callers.✏️ Suggested docstring tweak
- config_path: Ignored (kept for API compatibility, SAM2 uses Hydra config module) + config_path: Forwarded to algorithms.GDSegmenter for API compatibilitytests/rai_perception/services/test_weights.py-42-43 (1)
42-43: Patch target should be wheresubprocess.runis used.The patch should target
rai_perception.services.weights.subprocess.runinstead ofsubprocess.runto ensure the mock is applied where the function is imported/used, not where it's defined.Proposed fix
- with patch("subprocess.run", side_effect=mock_wget): + with patch("rai_perception.services.weights.subprocess.run", side_effect=mock_wget):Apply the same fix to all other
patch("subprocess.run", ...)calls in this file (lines 53-54, 73, 120).examples/manipulation-demo.py-41-54 (1)
41-54: Missingrclpy.shutdown()cleanup could cause resource leaks.
rclpy.init()is called at line 53, but there's no correspondingrclpy.shutdown()when the agent is done or on error. This can lead to resource leaks, especially ifcreate_agent()is called multiple times or if the program exits abnormally.Suggested approach
Consider adding cleanup in the
main()function or returning resources that need cleanup:def main(): - agent = create_agent() - messages: List[BaseMessage] = [] - - while True: - prompt = input("Enter a prompt: ") - messages.append(HumanMessage(content=prompt)) - output = agent.invoke({"messages": messages}) - output["messages"][-1].pretty_print() + try: + agent = create_agent() + messages: List[BaseMessage] = [] + + while True: + prompt = input("Enter a prompt: ") + messages.append(HumanMessage(content=prompt)) + output = agent.invoke({"messages": messages}) + output["messages"][-1].pretty_print() + except KeyboardInterrupt: + pass + finally: + rclpy.shutdown()src/rai_extensions/rai_perception/rai_perception/services/detection_service.py-75-77 (1)
75-77: Guard against empty class names in requests.
split(",")can yield empty strings from trailing, leading, or consecutive commas (e.g.,"dinosaur,"or"dinosaur,,dragon"). The current code strips whitespace but keeps empty strings, which propagate to the class dictionary and model. Filter empty strings to ensure only valid class names are processed.Proposed fix
- class_array = request.classes.split(",") - class_array = [class_name.strip() for class_name in class_array] + class_array = [c.strip() for c in request.classes.split(",") if c.strip()]src/rai_extensions/rai_perception/rai_perception/services/weights.py-112-118 (1)
112-118: Makeremove_weightsidempotent.
os.removeraises if the file is already gone, which can short‑circuit recovery paths. HandleFileNotFoundError.Proposed fix
def remove_weights(weights_path: Path): """Remove weights file. @@ Args: weights_path: Path to weights file to remove """ - os.remove(weights_path) + try: + os.remove(weights_path) + except FileNotFoundError: + passsrc/rai_extensions/rai_perception/rai_perception/__init__.py-21-36 (1)
21-36: Drop unusedE402noqa directives.
Ruff flags these as unused; removing them keeps lint clean.🧹 Proposed cleanup
-from .agents import GroundedSamAgent, GroundingDinoAgent # noqa: E402 -from .algorithms.point_cloud import depth_to_point_cloud # noqa: E402 -from .components.gripping_points import ( # noqa: E402 +from .agents import GroundedSamAgent, GroundingDinoAgent +from .algorithms.point_cloud import depth_to_point_cloud +from .components.gripping_points import ( @@ -from .components.topic_utils import ( # noqa: E402 +from .components.topic_utils import ( @@ -from .tools import GetDetectionTool, GetDistanceToObjectsTool # noqa: E402 -from .tools.gripping_points_tools import ( # noqa: E402 +from .tools import GetDetectionTool, GetDistanceToObjectsTool +from .tools.gripping_points_tools import (src/rai_extensions/rai_perception/rai_perception/components/service_utils.py-97-112 (1)
97-112: Explicitly mark unusedtimeout_secto satisfy lint.
Ruff flagstimeout_secas unused (ARG001); keep the signature but mark it intentionally unused.🧹 Proposed lint fix
def create_service_client( connector: ROS2Connector, service_type: Type, service_name: str, timeout_sec: float = 1.0, max_wait_time: float = 0.0, ) -> object: @@ - try: + _ = timeout_sec # kept for backward-compatible signature + try: wait_for_ros2_services(connector, [service_name], timeout=max_wait_time)src/rai_extensions/rai_perception/rai_perception/components/perception_presets.py-94-110 (1)
94-110: Fix return type hint to usetyping.Anyinstead of built-inany.
Line 94 uses lowercaseany, which type checkers will not recognize as a valid type. The correct annotation isDict[str, Any]. Note thatAnyis already imported on line 16.🛠️ Proposed fix
-def get_preset(preset_name: PresetName) -> Dict[str, any]: +def get_preset(preset_name: PresetName) -> Dict[str, Any]:
🧹 Nitpick comments (41)
src/rai_extensions/rai_perception/rai_perception/configs/detection_publisher.yaml (1)
9-10: Consider using>-or a plain scalar to avoid trailing newline.The YAML literal block scalar (
|) preserves trailing newlines, which may require trimming in the parsing code. Since the content is a single logical line, a folded/stripped scalar or plain quoted string would be cleaner:♻️ Suggested alternatives
Option 1: Plain scalar (simplest)
- detection_classes: | - person, cup, bottle, box, bag, chair, table, shelf, door, window, couch, sofa, bed, stove + detection_classes: "person, cup, bottle, box, bag, chair, table, shelf, door, window, couch, sofa, bed, stove"Option 2: Block scalar with strip indicator
- detection_classes: | + detection_classes: |- person, cup, bottle, box, bag, chair, table, shelf, door, window, couch, sofa, bed, stovesrc/rai_semap/rai_semap/ros2/config/detection_publisher.yaml (1)
5-5: Service endpoint updated; parameter naming reflects transitional state.The service endpoint change from
/grounding_dino_classifyto/detectionis correctly configured and the/detectionservice is implemented and available throughout the codebase. The parameterdino_servicenow points to this generic endpoint, creating a naming mismatch that could be addressed by renaming todetection_servicefor clarity. However, this would require coordinated updates across multiple configuration files and the detection_publisher component. Consider this change as part of a future refactoring effort to complete the API transition.src/rai_core/rai/tools/ros2/manipulation/custom.py (1)
280-280: Good deprecation notice with clear migration path.The decorator properly marks
GetObjectPositionsToolas deprecated and directs users to the newGetObjectGrippingPointsTool. Consider adding aversionparameter to track when the deprecation was introduced, which helps with release notes and migration timelines.♻️ Optional: Add version tracking to the deprecation
-@deprecated("Use GetObjectGrippingPointsTool from rai_perception instead") +@deprecated(version="0.x.0", reason="Use GetObjectGrippingPointsTool from rai_perception instead")Replace
0.x.0with the current package version.src/rai_extensions/rai_perception/pyproject.toml (1)
13-18: Python version constraint mismatch withrai_coredependency.
rai_perceptiondeclarespython = "^3.8"but depends onrai_corewhich requirespython = "^3.10, <3.13". This could cause confusing installation failures on Python 3.8 or 3.9.Consider aligning the Python constraint:
Proposed fix
[tool.poetry.dependencies] -python = "^3.8" +python = "^3.10, <3.13"tests/conftest.py (1)
21-30: Consider documenting available strategy values in the help text.The implementation is correct. For better discoverability, consider listing the valid strategy options in the help text.
Proposed improvement
def pytest_addoption(parser): parser.addoption( - "--strategy", action="store", default="centroid", help="Gripping point strategy" + "--strategy", + action="store", + default="centroid", + help="Gripping point strategy (e.g., centroid, top_plane, biggest_plane)", )src/rai_extensions/rai_perception/rai_perception/components/perception_utils.py (2)
32-45: Consider adding bounds validation for camera intrinsics extraction.The function assumes
camera_info.khas at least 6 elements (indices 0, 2, 4, 5). If a malformedCameraInfomessage is passed, this will raise anIndexError. Consider adding validation or documenting the precondition.Optional defensive check
def get_camera_intrinsics(camera_info: CameraInfo) -> Tuple[float, float, float, float]: """Extract camera intrinsics from CameraInfo message. Args: camera_info: Camera info message with intrinsics Returns: Tuple of (fx, fy, cx, cy) camera intrinsics + + Raises: + ValueError: If camera_info.k does not contain valid intrinsics """ + if len(camera_info.k) < 6: + raise ValueError("CameraInfo.k must contain at least 6 elements for intrinsics") fx = float(camera_info.k[0]) fy = float(camera_info.k[4]) cx = float(camera_info.k[2]) cy = float(camera_info.k[5]) return fx, fy, cx, cy
113-116: Inconsistent depth conversion - use the new helper.This function has inline depth conversion logic that duplicates
convert_depth_to_meters. For consistency and maintainability, consider using the helper here as well.Proposed fix
# Convert depth to meters (assuming depth image is in mm, adjust if needed) # Common depth encodings: 16UC1 (mm), 32FC1 (m) - if depth_image.encoding in ["16UC1", "mono16"]: - depth_value = depth_value / 1000.0 # mm to meters + # Create a single-element array to use the helper, then extract scalar + depth_arr = np.array([depth_value]) + depth_value = float(convert_depth_to_meters(depth_arr, depth_image)[0])Alternatively, extract the encoding check into a simpler helper or refactor
convert_depth_to_metersto also accept scalar values.tests/rai_perception/components/test_gripping_points_integration.py (1)
114-182: Avoid indefinite waits in the manual test.
wait_for_ros2_services/topicsdefault to an infinite wait. A bounded timeout makes the manual test fail fast when services are missing.⏱️ Add an explicit timeout
-def run_gripping_points_test( +def run_gripping_points_test( test_object: str, grasp: str, topics: dict[str, str], frames: dict[str, str], debug_enabled: bool = False, + timeout_sec: float = 30.0, ) -> None: @@ - wait_for_ros2_services(connector, REQUIRED_SERVICES) - wait_for_ros2_topics(connector, list(topics.values())) + wait_for_ros2_services(connector, REQUIRED_SERVICES, timeout=timeout_sec) + wait_for_ros2_topics(connector, list(topics.values()), timeout=timeout_sec) @@ def test_gripping_points_manipulation_demo(grasp) -> None: @@ run_gripping_points_test( test_object="cube", grasp=grasp, topics=MANIPULATION_DEMO_TOPICS, frames=MANIPULATION_DEMO_FRAMES, - debug_enabled=True, + debug_enabled=True, + timeout_sec=30.0, )tests/rai_perception/agents/test_base_vision_agent.py (1)
92-103: Consistent ROS2 setup/teardown pattern applied across test classes.The
setup_method/teardown_methodpattern for ROS2 context management is consistent with other test files (test_base_segmenter.py,test_base_boxer.py).The
try-except-passin teardown is intentional for robustness during test cleanup. However, per static analysis hints, consider logging exceptions rather than silently swallowing them, which could help diagnose intermittent test failures.♻️ Optional: Log exceptions in teardown for better diagnostics
def teardown_method(self): """Clean up ROS2 context after each test.""" try: if rclpy.ok(): - rclpy.shutdown() + time.sleep(0.1) # Allow pending callbacks to complete + rclpy.shutdown() except Exception: - pass + pass # Intentionally swallowed - cleanup should not fail testsOr add logging:
except Exception as e: import logging logging.debug(f"ROS2 shutdown during teardown: {e}")src/rai_extensions/rai_perception/rai_perception/components/visualization_utils.py (2)
32-60: Code duplication:_quaternion_to_rotation_matrixis duplicated fromgripping_points.py.This function is an exact duplicate of the implementation in
src/rai_extensions/rai_perception/rai_perception/components/gripping_points.py(lines 340-360). Consider extracting it to a shared math utilities module to maintain DRY principles.♻️ Suggested refactor
Create a shared module like
math_utils.py:# In rai_perception/components/math_utils.py def quaternion_to_rotation_matrix(qx: float, qy: float, qz: float, qw: float) -> np.ndarray: ...Then import from both
gripping_points.pyandvisualization_utils.py.
181-185: Silent failure on transform errors may hide issues in downstream processing.Catching a broad
Exceptionand returning original (untransformed) points could lead to incorrect visualizations without any indication to the caller that the transform failed. The error is logged, but callers have no programmatic way to detect the failure.Consider either:
- Re-raising after logging (if callers should handle failures)
- Returning a tuple
(points, success: bool)to indicate transform status- Keeping current behavior if silent fallback is intentional (document this clearly)
examples/manipulation-demo-v1.py (1)
25-33: Consider adding graceful exit handling and basic error handling.The infinite loop with
input()will raiseKeyboardInterrupton Ctrl+C without cleanup. For a demo script, consider wrapping with try-except for cleaner exit:♻️ Suggested improvement
def main(): agent, _ = create_agent() messages: List[BaseMessage] = [] - while True: - prompt = input("Enter a prompt: ") - messages.append(HumanMessage(content=prompt)) - output = agent.invoke({"messages": messages}) - output["messages"][-1].pretty_print() + try: + while True: + prompt = input("Enter a prompt: ") + if not prompt.strip(): + continue + messages.append(HumanMessage(content=prompt)) + output = agent.invoke({"messages": messages}) + output["messages"][-1].pretty_print() + except KeyboardInterrupt: + print("\nExiting...")src/rai_extensions/rai_perception/rai_perception/scripts/run_perception_services.py (1)
23-45: Consider adding cleanup handling for initialization failures.If
DetectionServiceorSegmentationServiceinitialization fails (e.g., model loading error),rclpy.shutdown()won't be called, potentially leaving ROS2 in an unclean state.♻️ Suggested improvement
def main(): rclpy.init() - - # Create ROS2 connectors for each service - # TODO(juliaj): Re-evaluate executor_type choice (single_threaded vs multi_threaded) - # Current: single_threaded for consistency with BaseVisionService pattern - # Consider: multi_threaded if concurrent request handling is needed - detection_connector = ROS2Connector( - "detection_service", executor_type="single_threaded" - ) - segmentation_connector = ROS2Connector( - "segmentation_service", executor_type="single_threaded" - ) - - # Services read model_name from ROS2 params (defaults: "grounding_dino", "grounded_sam") - detection_service = DetectionService(ros2_connector=detection_connector) - segmentation_service = SegmentationService(ros2_connector=segmentation_connector) - - detection_service.run() - segmentation_service.run() - - wait_for_shutdown([detection_service, segmentation_service]) - rclpy.shutdown() + try: + # Create ROS2 connectors for each service + # TODO(juliaj): Re-evaluate executor_type choice (single_threaded vs multi_threaded) + # Current: single_threaded for consistency with BaseVisionService pattern + # Consider: multi_threaded if concurrent request handling is needed + detection_connector = ROS2Connector( + "detection_service", executor_type="single_threaded" + ) + segmentation_connector = ROS2Connector( + "segmentation_service", executor_type="single_threaded" + ) + + # Services read model_name from ROS2 params (defaults: "grounding_dino", "grounded_sam") + detection_service = DetectionService(ros2_connector=detection_connector) + segmentation_service = SegmentationService(ros2_connector=segmentation_connector) + + detection_service.run() + segmentation_service.run() + + wait_for_shutdown([detection_service, segmentation_service]) + finally: + rclpy.shutdown()tests/rai_perception/algorithms/test_point_cloud.py (1)
65-83: Make the coordinate test order-agnostic.This test currently assumes a specific point ordering; if the implementation changes its flattening order, it will become brittle. Consider comparing sorted arrays instead.
♻️ Proposed refactor (order-agnostic comparison)
- # Verify coordinates for each pixel - for i, point in enumerate(points): - u, v = i % 3, i // 3 - expected_x = (u - cx) * depth_value / fx - expected_y = (v - cy) * depth_value / fy - - np.testing.assert_almost_equal(point[0], expected_x, decimal=5) - np.testing.assert_almost_equal(point[1], expected_y, decimal=5) - np.testing.assert_almost_equal(point[2], depth_value, decimal=5) + expected = [] + for v in range(3): + for u in range(3): + expected.append( + [ + (u - cx) * depth_value / fx, + (v - cy) * depth_value / fy, + depth_value, + ] + ) + expected = np.array(expected, dtype=np.float32) + + # Order-agnostic comparison + points_sorted = points[np.lexsort((points[:, 2], points[:, 1], points[:, 0]))] + expected_sorted = expected[ + np.lexsort((expected[:, 2], expected[:, 1], expected[:, 0])) + ] + np.testing.assert_allclose(points_sorted, expected_sorted, rtol=1e-5, atol=1e-5)src/rai_extensions/rai_perception/rai_perception/models/detection.py (1)
57-74: Consider aligning return type with segmentation module for API consistency.The segmentation module's
get_modelreturnsTuple[Type, str | None]while this returnsTuple[Type, str]. If future detection models might not require config paths, consider using the nullable type for consistency across model registries.tests/rai_perception/tools/test_pcl_detection_tools.py (3)
19-24: Simplify the ROS2 import check.The
noqadirectives are flagged as unused by static analysis. The import check can be simplified:♻️ Suggested simplification
try: - import rclpy # noqa: F401 - - _ = rclpy # noqa: F841 + import rclpy except ImportError: pytest.skip("ROS2 is not installed", allow_module_level=True)
98-138: Consider seeding random number generator for test reproducibility.The test uses
np.random.normalwithout a seed, which could theoretically cause flaky tests. While the assertions appear tolerant enough, seeding ensures deterministic behavior:♻️ Optional fix for reproducibility
def test_point_cloud_filter(): """Test point cloud filtering strategies.""" + np.random.seed(42) # Ensure reproducible test data # Create test data with noise points main_cluster = np.random.normal([0, 0, 0], 0.1, (50, 3)).astype(np.float32)
170-172: Prefix unused parameter with underscore to indicate intentional non-use.The
obj_nameparameter is needed to match the mocked method's signature but isn't used. Prefixing with underscore silences the linter and documents the intent:♻️ Suggested fix
- def slow_operation(obj_name): + def slow_operation(_obj_name): time.sleep(2.0) # Longer than timeout return []tests/rai_perception/components/test_topic_utils.py (1)
17-22: Simplify the ROS2 import check (same as other test file).♻️ Suggested simplification
try: - import rclpy # noqa: F401 - - _ = rclpy # noqa: F841 + import rclpy except ImportError: pytest.skip("ROS2 is not installed", allow_module_level=True)src/rai_extensions/rai_perception/rai_perception/services/base_vision_service.py (1)
93-97: Consider extracting the deferred import to reduce duplication.The
get_param_valueimport is duplicated in both_initialize_model_from_registry(line 93) and_get_service_name(line 119). While deferred imports help avoid circular dependencies, the duplication could be consolidated.Optional: Extract to a module-level lazy import helper or import once at method entry
+ def _get_param_value(self, name: str, default=None): + """Get ROS2 parameter value with lazy import.""" + from rai.communication.ros2 import get_param_value + return get_param_value(self.ros2_connector.node, name, default=default) + def _initialize_model_from_registry( self, get_model_func, default_model_name: str, model_type_name: str ): ... - from rai.communication.ros2 import get_param_value - - model_name = get_param_value( - self.ros2_connector.node, "model_name", default=default_model_name - ) + model_name = self._get_param_value("model_name", default=default_model_name)Also applies to: 119-123
docs/extensions/rethinking_usability.md (1)
251-260: Add language specifier to fenced code block.The code block describing the workflow lacks a language specifier. Adding one improves rendering in documentation viewers.
-``` +```text 1. list_available_models() → ["grounding_dino", "yolo", ...]src/rai_core/rai/tools/timeout.py (2)
49-49: Use explicitstr | Nonetype annotation instead of implicit Optional.Per PEP 484,
timeout_message: str = Noneimplicitly creates an Optional type which is discouraged. Use explicitstr | Nonefor clarity.Proposed fix
-def timeout(seconds: float, timeout_message: str = None) -> Callable[[F], F]: +def timeout(seconds: float, timeout_message: str | None = None) -> Callable[[F], F]:-def timeout_method(seconds: float, timeout_message: str = None) -> Callable[[F], F]: +def timeout_method(seconds: float, timeout_message: str | None = None) -> Callable[[F], F]:Also applies to: 103-103
91-96: Chain exceptions usingraise ... fromfor better tracebacks.When raising
RaiTimeoutErrorin the except clause, chain it withfrom err(orfrom Noneto suppress the original) to provide clearer exception context and satisfy B904.Proposed fix for both decorators
except concurrent.futures.TimeoutError: message = ( timeout_message or f"Function '{func.__name__}' timed out after {seconds} seconds" ) - raise RaiTimeoutError(message) + raise RaiTimeoutError(message) from Noneexcept concurrent.futures.TimeoutError: message = ( timeout_message or f"Method '{func.__name__}' of {self.__class__.__name__} timed out after {seconds} seconds" ) - raise RaiTimeoutError(message) + raise RaiTimeoutError(message) from NoneAlso applies to: 150-155
tests/tools/test_timeout.py (1)
1-61: Add test coverage for the standalonetimeoutdecorator.The tests thoroughly cover
timeout_methodbehavior, but the standalonetimeoutdecorator (for regular functions) lacks test coverage. Both decorators are implemented in the module but serve different purposes:timeoutfor functions andtimeout_methodfor instance methods, with different default error messages.Consider adding a test for the standalone
timeoutdecorator:Suggested test code
from rai.tools.timeout import RaiTimeoutError, timeout, timeout_method `@timeout`(0.5) def slow_function(): time.sleep(1.0) return "should not reach here" def test_timeout_decorator_raises_timeout_error(): with pytest.raises(RaiTimeoutError) as exc_info: slow_function() assert "slow_function" in str(exc_info.value) assert "0.5 seconds" in str(exc_info.value)tests/rai_perception/tools/test_segmentation_tools.py (1)
223-326: Consider removing duplicate parameter setup in the grab‑point test.
conversion_ratiois set twice, which is redundant and makes the test longer than needed.♻️ Optional cleanup
- # Set ROS2 parameters - mock_connector.node.set_parameters( - [ - Parameter( - "conversion_ratio", rclpy.parameter.Parameter.Type.DOUBLE, 0.001 - ), - ] - )src/rai_extensions/rai_perception/rai_perception/algorithms/point_cloud.py (1)
40-48: Guard against invalid intrinsics and non‑finite depth values.If
fx/fyare zero or depth contains non‑finite values, the output can includeinf/nanpoints that pass thez > 0filter. Consider validating intrinsics and filtering finite points.🛠️ Suggested hardening
def depth_to_point_cloud( depth_image: NDArray[np.float32], fx: float, fy: float, cx: float, cy: float ) -> NDArray[np.float32]: @@ - height, width = depth_image.shape + if fx == 0 or fy == 0: + raise ValueError("fx and fy must be non-zero") + height, width = depth_image.shape @@ - points = np.stack((x, y, z), axis=-1).reshape(-1, 3) - points = points[points[:, 2] > 0] + points = np.stack((x, y, z), axis=-1).reshape(-1, 3) + valid = (points[:, 2] > 0) & np.isfinite(points).all(axis=1) + points = points[valid] return points.astype(np.float32, copy=False)tests/rai_perception/algorithms/test_base_boxer.py (1)
43-55: Consider more robust ROS2 teardown.The
time.sleep(0.1)in teardown is a workaround for thread cleanup timing. While functional, this could be flaky in CI environments with varying loads.The current implementation works but the sleep-based approach is inherently racy. Consider documenting why the sleep is needed or investigating if there's a more deterministic cleanup method.
src/rai_extensions/rai_perception/rai_perception/services/segmentation_service.py (1)
115-128: Consider tracking partial mask failures.When individual mask processing fails (line 122-126), the error is logged but the response still returns successfully processed masks. This is generally good for resilience, but callers have no way to know that some masks failed.
If partial failures need to be surfaced, consider either:
- Adding a status field to the response message.
- Logging a summary at the end (e.g., "Processed X/Y masks successfully").
Current behavior is acceptable if callers can tolerate partial results silently.
src/rai_extensions/rai_perception/rai_perception/tools/gdino_tools.py (3)
226-234: Avoid mutable class-level defaults forrequired_services.
UseClassVar(or tuples) so this metadata isn’t treated as a mutable instance field. AddClassVarto the typing imports.♻️ Proposed fix
- required_services: list[dict[str, str]] = [ + required_services: ClassVar[list[dict[str, str]]] = [
284-304: Also markrequired_services/pipeline_stagesas ClassVar.
Same mutability concern here; these are metadata and shouldn’t be mutable instance fields.♻️ Proposed fix
- required_services: list[dict[str, str]] = [ + required_services: ClassVar[list[dict[str, str]]] = [ @@ - pipeline_stages: list[dict[str, str]] = [ + pipeline_stages: ClassVar[list[dict[str, str]]] = [
405-461: Consider inheriting service info helpers from the base class.
get_service_infoandcheck_service_dependenciesduplicate the base implementation; removing them would reduce drift while keepingget_pipeline_infohere.src/rai_extensions/rai_perception/rai_perception/services/detection_service.py (1)
45-52: Prefer a shared constant for the default weights root.
Using a centralized constant avoids evaluatingPath.home()in the signature and keeps defaults consistent.♻️ Proposed fix
- weights_root_path: str | Path = Path.home() / Path(".cache/rai"), + weights_root_path: str | Path = BaseVisionService.DEFAULT_WEIGHTS_ROOT_PATH,src/rai_extensions/rai_perception/rai_perception/algorithms/boxer.py (2)
50-52:class_dictis unused.
Either remove it or use it to map class IDs to avoid confusing callers (update call sites accordingly).♻️ Possible simplification
- def to_detection_msg( - self, class_dict: Dict[str, int], timestamp: Time - ) -> Detection2D: + def to_detection_msg(self, timestamp: Time) -> Detection2D:
56-65: Resolve the timestamp type ambiguity.
Consider widening the type hint (e.g., a union) or normalizing at call sites so this TODO doesn’t linger.Want me to draft the signature update and call‑site fixes?
src/rai_extensions/rai_perception/rai_perception/services/weights.py (1)
64-109: Add a timeout to avoid hanging downloads.A stalled
wgetcan block startup indefinitely; consider a subprocess timeout (and optionally retries) so failures surface promptly. Please verify this aligns with the runtime image’swgetavailability and timeout expectations.Proposed tweak
subprocess.run( [ "wget", weights_url, "-O", str(weights_path), "--progress=dot:giga", ], check=True, capture_output=True, text=True, + timeout=600, )src/rai_extensions/rai_perception/rai_perception/algorithms/segmenter.py (1)
43-78: Guard Hydra global initialization.
GlobalHydra.instance().clear()resets global state for the whole process; if other components rely on Hydra, this can cause interference. Consider initializing only when needed (or isolating this init) and verify behavior against the Hydra version in use.tests/rai_perception/algorithms/test_base_segmenter.py (1)
45-57: Avoid swallowing teardown errors silently.Catching broad exceptions and
passcan hide flaky ROS2 teardown issues; consider logging for visibility. Please verify against your pytest/ROS2 noise tolerance.Proposed tweak
import time +import logging @@ def teardown_method(self): """Clean up ROS2 context after each test.""" try: if rclpy.ok(): time.sleep(0.1) rclpy.shutdown() - except Exception: - pass + except Exception: + logging.getLogger(__name__).exception("rclpy shutdown failed")src/rai_extensions/rai_perception/rai_perception/agents/grounding_dino.py (1)
24-64: AvoidPath.home()in default arguments.Default-arg evaluation happens at import time; a
Nonedefault (or module constant) is safer and easier to patch in tests.♻️ Proposed refactor
GDINO_NODE_NAME = "grounding_dino" GDINO_SERVICE_NAME = "grounding_dino_classify" +DEFAULT_WEIGHTS_ROOT = Path.home() / Path(".cache/rai") class GroundingDinoAgent(BaseAgent): @@ def __init__( self, - weights_root_path: str | Path = Path.home() / Path(".cache/rai"), + weights_root_path: str | Path | None = None, ros2_name: str = GDINO_NODE_NAME, ): @@ - self.ros2_connector, self._service = create_service_wrapper( + if weights_root_path is None: + weights_root_path = DEFAULT_WEIGHTS_ROOT + self.ros2_connector, self._service = create_service_wrapper( DetectionService, ros2_name, "grounding_dino", GDINO_SERVICE_NAME, weights_root_path, )src/rai_extensions/rai_perception/rai_perception/tools/segmentation_tools.py (1)
153-168: Avoid pollingget_future_resultin a loop.Each call registers another callback and can spam warnings. A single wait with an explicit timeout and graceful fallback is safer.
♻️ Proposed refactor
- resolved = None - while rclpy.ok(): - resolved = get_future_result(future) - if resolved is not None: - break + resolved = get_future_result(future, timeout_sec=60.0) + if resolved is None: + logger.warning("Detection service timed out") + return "", {"segmentations": []} @@ - ret = [] - while rclpy.ok(): - resolved = get_future_result(future) - if resolved is not None: - for img_msg in resolved.masks: - ret.append(convert_ros_img_to_base64(img_msg)) - break + ret = [] + resolved = get_future_result(future, timeout_sec=60.0) + if resolved is None: + logger.warning("Segmentation service timed out") + return "", {"segmentations": []} + for img_msg in resolved.masks: + ret.append(convert_ros_img_to_base64(img_msg))src/rai_extensions/rai_perception/rai_perception/tools/gripping_points_tools.py (2)
93-109: Mark class-level constants asClassVarto keep them out of Pydantic fields.♻️ Proposed refactor
-from typing import Any, Callable, Dict, Optional, Type +from typing import Any, Callable, ClassVar, Dict, Optional, Type @@ - pipeline_stages: list[dict[str, str]] = [ + pipeline_stages: ClassVar[list[dict[str, str]]] = [ @@ - required_services: list[dict[str, str]] = [ + required_services: ClassVar[list[dict[str, str]]] = [Also applies to: 112-125
167-205: Silence unusedstage_timeresults.♻️ Proposed refactor
- pcl, stage_time = self._run_stage( + pcl, _stage_time = self._run_stage( @@ - pcl_filtered, stage_time = self._run_stage( + pcl_filtered, _stage_time = self._run_stage( @@ - gripping_points, stage_time = self._run_stage( + gripping_points, _stage_time = self._run_stage(
src/rai_extensions/rai_perception/rai_perception/components/detection_publisher.py
Show resolved
Hide resolved
src/rai_extensions/rai_perception/rai_perception/components/gripping_points.py
Show resolved
Hide resolved
src/rai_extensions/rai_perception/rai_perception/components/gripping_points.py
Show resolved
Hide resolved
src/rai_extensions/rai_perception/rai_perception/components/service_utils.py
Show resolved
Hide resolved
src/rai_extensions/rai_perception/rai_perception/components/topic_utils.py
Show resolved
Hide resolved
Replace manual conversion_ratio with ROS2 standard encoding-aware depth conversion in GetObjectGrippingPointsTool. Automatically handles 16UC1/mono16 (mm) and 32FC1 (m) encodings based on image metadata.
…to run both versions
@jmatejcz thanks for help look into this. The current code changes are back-compat by default. Once we have this merged, I will look into migrate existing code to the new service names. |
src/rai_extensions/rai_perception/rai_perception/components/exceptions.py
Outdated
Show resolved
Hide resolved
@jmatejcz, thanks for looking into this. The testing I have done is listed in the PR description. A few additional things perhaps
|
Purpose
Proposed Changes
Code refactor based on Rethinking Usability and API Usability Design Considerations.
There are breaking changes introduced, see section below for details. To reduce the impact on applications using the old service names, a legacy service names flag (
enable_legacy_service_names) has been introduced and the default behavior is backward compatible (defaults totrue). For new applications using only the new service names (/detection,/segmentation), you can disable legacy names by setting the flag tofalse:Via launch file argument:
Via environment variable:
Via ROS2 parameter (in code):
Migration Guide
Issues
Testing
To start manipulation-streamlit with both versions
Summary by CodeRabbit
New Features
Documentation
Bug Fixes & Improvements
✏️ Tip: You can customize this high-level summary in your review settings.