Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(autoware_tensorrt_bevdet): add new 3d object detection method #7956

Open
wants to merge 85 commits into
base: main
Choose a base branch
from

Conversation

cyn-liu
Copy link
Contributor

@cyn-liu cyn-liu commented Jul 11, 2024

Description

Integrating BEVDet into Autoware for 3D object detection based on multi-view images.

Related links

Issue Link

How was this PR tested?

This PR has been tested on local environment.
env1:

RTX 3080
docker image: `ghcr.io/autowarefoundation/autoware-universe:humble-20231001-cuda`
CUDA11.6
TensorRT8.4.2

env2:

RTXA1000 `Dell notebook computer`
docker image: ghcr.io/autowarefoundation/autoware:20240612-devel-cuda-amd64
CUDA==12.3
libnvinfer==8.6.1.6

Notes for reviewers

Test using a bag containing Nuscenes data.
Note: The frame_id of /lidar_top in this bag is map
onnx model file
bag

Interface changes

Add a new perception_mode, If set perception_mode = camera, the detector will launch tesnsorrt_bevdet node.

Topic changes

Additions and removals

Change type Topic Type Topic Name Message Type Description
Added Sub ~/input/topic_cloud sensor_msgs::msg::PointCloud2 used for time alignment and display
Added Sub ~/input/topic_img_fl sensor_msgs::msg::Image input front_left camera image
Added Sub ~/input/topic_img_f sensor_msgs::msg::Image input front camera image
Added Sub ~/input/topic_img_fr sensor_msgs::msg::Image input front_right camera image
Added Sub ~/input/topic_img_bl sensor_msgs::msg::Image input back_left camera image
Added Sub ~/input/topic_img_b sensor_msgs::msg::Image input back camera image
Added Sub ~/input/topic_img_br sensor_msgs::msg::Image input back_right camera image
Added Pub ~/output/painting_cloud sensor_msgs::msg::PointCloud2 output pointcloud just used for display
Added Pub ~/output/boxes autoware_perception_msgs::msg::DetectedObjects detected objects

ROS Parameter Changes

Additions and removals

Change type Parameter Name Type Default Value Description
Added bev_detection_model string bevdet Set BEV detection model type

Modifications

Version Parameter Name Type Default Value Description
Old perception_mode string lidar Set perception mode
New perception_mode string lidar Add a new perception mode of camera

Effects on system behavior

None.

@github-actions github-actions bot added type:documentation Creating or refining documentation. (auto-assigned) component:perception Advanced sensor data processing and environment understanding. (auto-assigned) component:launch Launch files, scripts and initialization tools. (auto-assigned) labels Jul 11, 2024
Copy link

github-actions bot commented Jul 11, 2024

Thank you for contributing to the Autoware project!

🚧 If your pull request is in progress, switch it to draft mode.

Please ensure:

@kminoda kminoda added the tag:run-build-and-test-differential Mark to enable build-and-test-differential workflow. (used-by-ci) label Jul 11, 2024
Copy link
Contributor

@kminoda kminoda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool PR 🚀

Let me provide some minor comments first. Also, please make sure that all the CIs are passing before opening the PR 🙏

perception/tensorrt_bevdet/package.xml Outdated Show resolved Hide resolved
perception/tensorrt_bevdet/include/bevdet_node.hpp Outdated Show resolved Hide resolved
@cyn-liu cyn-liu linked an issue Jul 16, 2024 that may be closed by this pull request
7 tasks
@kminoda
Copy link
Contributor

kminoda commented Jul 22, 2024

(Friendly ping 🙏 ) Would you mention us once all the CIs are fixed?

@technolojin
Copy link
Contributor

I could see two required checks are not passed.

  1. cppcheck-differential > Details > Show cppcheck-report result
Run cat cppcheck-report.txt
Error: /home/runner/work/autoware.universe/autoware.universe/perception/tensorrt_bevdet/src/bevdet.cpp:167:13: performance: Range variable 'task_factors' should be declared as const reference. [iterateByValue]
  for (auto task_factors : nms_factor_temp) {
            ^
Error: /home/runner/work/autoware.universe/autoware.universe/perception/tensorrt_bevdet/src/data.cpp:37:20: performance: Range variable 'name' should be declared as const reference. [iterateByValue]
  for (std::string name : cams_name) {
                   ^
  1. pre-commit.ci -pr > Details
fix include guard........................................................Failed
- hook id: ros-include-guard
- exit code: 1

No include guard in perception/tensorrt_bevdet/include/gatherbev_plugin.hpp
No include guard in perception/tensorrt_bevdet/include/bevpool_plugin.hpp
No include guard in perception/tensorrt_bevdet/include/preprocess_plugin.hpp
No include guard in perception/tensorrt_bevdet/include/bevdet_node.hpp
No include guard in perception/tensorrt_bevdet/include/alignbev_plugin.hpp

sort package.xml.........................................................Passed
shellcheck...............................................................Passed
shfmt....................................................................Passed
isort....................................................................Passed
black....................................................................Passed
clang-format.............................................................Passed
cpplint..................................................................Failed
- hook id: cpplint
- exit code: 1

perception/tensorrt_bevdet/include/bevdet_node.hpp:0:  No copyright message found.  You should have a line: "Copyright [year] <Copyright Owner>"  [legal/copyright] [5]
perception/tensorrt_bevdet/include/bevdet_node.hpp:86:  Add #include <vector> for vector<>  [build/include_what_you_use] [4]
perception/tensorrt_bevdet/include/bevdet_node.hpp:113:  Add #include <memory> for shared_ptr<>  [build/include_what_you_use] [4]
Done processing perception/tensorrt_bevdet/include/bevdet_node.hpp
perception/tensorrt_bevdet/include/cpu_jpegdecoder.hpp:0:  No copyright message found.  You should have a line: "Copyright [year] <Copyright Owner>"  [legal/copyright] [5]
Done processing perception/tensorrt_bevdet/include/cpu_jpegdecoder.hpp
perception/tensorrt_bevdet/include/gatherbev_plugin.hpp:45:  Single-parameter constructors should be marked explicit.  [runtime/explicit] [5]
Done processing perception/tensorrt_bevdet/include/gatherbev_plugin.hpp
perception/tensorrt_bevdet/include/preprocess.hpp:0:  No copyright message found.  You should have a line: "Copyright [year] <Copyright Owner>"  [legal/copyright] [5]
Done processing perception/tensorrt_bevdet/include/preprocess.hpp
perception/tensorrt_bevdet/src/bevdet.cpp:0:  No copyright message found.  You should have a line: "Copyright [year] <Copyright Owner>"  [legal/copyright] [5]
perception/tensorrt_bevdet/src/bevdet.cpp:190:  Using C-style cast.  Use static_cast<float>(...) instead  [readability/casting] [4]

...

@technolojin technolojin self-assigned this Jul 26, 2024
@kminoda
Copy link
Contributor

kminoda commented Jul 29, 2024

@cyn-liu As this PR is stale for a week, from the maintenance perspective, let me make this PR a draft for now. Feel free to re-open the PR once all the CIs are fixed and ready to be reviewed. Thank you for your understanding, and we are looking forward to reviewing your PR.

@kminoda kminoda marked this pull request as draft July 29, 2024 02:27
feat(camera_info,tf): use camera info and TF instead of config file
Copy link
Contributor

@kminoda kminoda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mojomex mojomex marked this pull request as draft September 6, 2024 06:47
@cyn-liu cyn-liu marked this pull request as ready for review September 18, 2024 09:30
ranks_depth_ptr.reset(new int[valid_feat_num_], std::default_delete<int[]>());
ranks_feat_ptr.reset(new int[valid_feat_num_], std::default_delete<int[]>());
interval_starts_ptr.reset(new int[interval_starts_host.size()], std::default_delete<int[]>());
interval_lengths_ptr.reset(new int[interval_lengths_host.size()], std::default_delete<int[]>());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:launch Launch files, scripts and initialization tools. (auto-assigned) component:perception Advanced sensor data processing and environment understanding. (auto-assigned) tag:require-cuda-build-and-test tag:run-build-and-test-differential Mark to enable build-and-test-differential workflow. (used-by-ci) type:documentation Creating or refining documentation. (auto-assigned)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implementing BEVDet in Autoware
8 participants