Mission Planner with Locally-generated, Sensor-based Maps

[simon-eisenmann-driveblocks]

Mission Planner with Locally-generated, Sensor-based Maps

The following proposal is a joint project of TIER IV and driveblocks.

We are exploring an innovative approach to enhance the autonomy of vehicles using the Autoware software framework. Our proposal involves the development of a local mission and behavior planner utilizing local maps, which are created based on the sensor inputs from cameras and LiDARs. For this proposal, we assume these sensor-based maps to be readily available. They can be used for full self-driving when no HD maps are available.

The following screenshot shows camera images from a Japan Taxi vehicle (images on the bottom) recorded by two cameras mounted on the vehicle's rooftop. The resulting local map, which consists of four road segments that fit into the available highway lanes and is computed based on the sensor inputs, is visualized at the top of the screenshot. The camera images on the bottom are overlayed with the results of the line detection and with the projected local map.
Please note that we assume a local map will be given for our proposal.

Motivation

Currently, the mission and behavior planning modules in Autoware rely on global, static HD maps, which can be expensive to create and maintain.
We propose to enable autonomous operation without the need for HD maps by allowing Autoware to use locally created, sensor-based maps.
When operating with local maps, the navigation goal may not always be present or visible in a locally available map snippet—in contrast to HD maps. Therefore, we would like to propose a concept for new local mission and behavior planning modules that can support a locally feasible driving behavior when no HD map is available.
The current architecture of Autoware is kept unchanged, but the new local mission and behavior planning components shall form the basis for a new HD map-free planning component. In the future, the HD map-free planning component can be unified with the current Autoware planning component to enable seamless transitions between scenarios (e.g., urban driving with an HD map to highway driving with a local map) or to combine HD maps and local, sensor-based ones.

In this discussion, we would like to collect your feedback and ideas on implementing a local mission and behavior planner for local maps and their integration into Autoware.

Use Cases

In a first step, we would like to target highway use cases, explicitly supporting the scenarios of

• lane keeping,
• lane changing,
• taking the next exit.

The following section proposes a software architecture to handle these scenarios with a local map.

Proposed Architecture

In the following diagram, we visualize a possible software architecture. From top to bottom, we include a custom local road provider that computes the road segments based on sensor inputs. A local map provider node transforms the road segments into an Autoware local map message, which we must implement. Therein, the detected road segments are converted into driving corridors that comprise lane centerlines, left and right boundaries, and semantic types, e.g., road, highway exit, or shoulder lane. The map provider node also enriches the driving corridors with navigation information from OpenStreetMaps or detected road signs. The new Autoware local mission planner picks up the local map representation and a local goal. It extracts the mission lanes containing the target lane, drivable lanes, and a spatial deadline parameter indicating when the mission goal must be reached. The mission goal is a new Autoware message that defines the mission type (e.g., lane keeping, lane change, take next exit) and includes the mentioned spatial deadline parameter. This parameter can be set to infinity if it doesn't apply to any specific mission (like lane keeping). The new local behavior planner computes a rough, collision-free reference trajectory to the target lane while adhering to the constraints imposed by the spatial deadline parameter, the drivable lanes, and the surrounding objects. An additional motion planner could smooth the reference trajectory before forwarding it to the controller.

The data flow from the Human Machine Interface (HMI) to the controller follows a structured path, which will be introduced in the next section.

Example Use Cases

First, let us introduce the data flow from the goal information source to the control output. The global goal can be identified algorithmically by a goal planner or be set by the safety driver via the Autoware HMI. The safety driver would initiate a particular behavior by setting a ROS mission parameter (given in the table below, first column) via the terminal/a GUI, which is then converted into a local goal message of type autoware_auto_planning_msgs::msg::LocalMission. The safety driver could initiate, e.g., the missions lane_keeping, lane_change, or take_next_exit.

Local Mission Planner

A set of rules is defined for each mission type to select the target and drivable lanes, described in the message type autoware_auto_planning_msgs::msg::MissionLanes from the local map.

field/mission	lane_keeping	lane_change	take_next_exit
target_lane	ego lane	neighboring lane	outmost lane until exit is available in local road model
drivable_lanes	directly neighboring lanes	ego lane and directly neighboring lanes	ego lane and neighboring lanes towards exit
deadline	infinity	available distance to finish lane change	distance to the exit

If a local mission does not clearly indicate its meaning in the available local map, we propose continuing with the currently active mission type or postponing this request for a maximum distance of a few hundred meters. An example is a mission request to stop on the shoulder lane even if no such lane is available.

Local Behavior Planner

Based on the MissionLanes, the behavior planner computes a collision-free trajectory such that the ego drives along or switches to the target lane while following the constraints imposed by the deadline parameter, the drivable lanes, and the surrounding objects. If necessary, the trajectory could cross the drivable lanes to, e.g., avoid an obstacle or cross multiple lanes.

Future Directions

In the future, combining the information from local with global maps (such as OpenStreetMap or HD Maps) will become necessary. Through this, navigation information can be fused with our local, sensor-based map to combine both their advantages: global navigation knowledge and local topicality. A global navigation information source and an advanced local behavior planner enable the omission of human mission input. Additionally, exploring supplementary sources like street sign detectors can enrich the information pool, facilitating comprehensive decision-making processes. As the introduction states, the mapless architecture will function as a separate planning component. Our ultimate vision involves combining this component into a single architecture with the existing Autoware modules to seamlessly transition from HD to local maps or combine both into a single environment representation if desired.

Seeking Feedback

We're excited about this idea's potential and would love to hear your thoughts, feedback, and suggestions. Are the architecture and the defined interfaces capable of handling the use cases and missions that come to your mind regarding local maps?

Please share your insights, experiences, or any questions you have. We look forward to collaborating with the community to make local maps available to the Autoware software stack.

Thank you for your interest and contributions!

[Owen-Liuyuxuan]

Great job! Very excited to see progress in sensor-based maps. I am welcoming to see this progressive approach to include local-map-based planning step by step.

I would like to share some of my reactions to the architecture and would love to learn from your response :).

1. Perception Modules and Computation: It seems that we are setting object detection and local road provider as separate nodes at least for now. I would love too see shared-backbone methods (BEV for example) to generate object detection and maps in one model. Also turning signal detection / road sign recognition are important in highway driving so there may be a lot of computations too.
2. Map APIs: When talking about highway navigation, I would naturally think of openpilot. Will Autoware consider receiving navigation commands from map APIs (or we simulate what map APIs do with a goal planner and OSM)? Also will we about to receive speed limit / road construction notification from map APIs?
3. Planning/Control in relative coordinate: When a human is driving on the highway, I feel like we are using a "relative coordinate" to make planning/control decisions. If we would like to make a lane change with a "global coordinate", the path will be extremely long and flat, personally I doubt this could be performed well in either planning/control. I would like to know if there are any ideas or reference knowledge on this problem.
4. State Transition: One critical problem will be about merging these two kinds of maps. We may need to accurately localizing the current position w.r.t. the HDMap when we are transferring from sensor map -> HD Map to make sure the lane alignment is correct. Another critical problem will be the state transition of PC modules and I would really like to learn how Autoware is considered to do this.

[maxime-clem]

Thank you for your nice comments and suggestions.

1. You are right that we can consider a single model that generates the local map along with object detection. I think this can be achieved with the proposed architecture with minimal changes. E.g., by combining the Local Road Provider and Local Map Provider into a single black-box that outputs Object Detection in addition to the local map.
2. I do not have experience using openpilot and I could not find any document about a map API. Can you explain your idea in more details ?
3. You are correct and the local map in the proposed architecture is most likely going to be expressed in a frame relative to the ego vehicle. Planning/control will then use trajectories in relative frame as well (e.g., base_link).
4. We thought about this transition and it is a difficult problem. At first, we can consider the 2 architectures running in parallel (local map architecture + HD map architecture) and just switching between a local trajectory (e.g., base_link frame) to a global trajectory (map frame), allowing the same controller to be used by both architecture. In the future, we want to consider unifying both architectures but this will require a very general map format that can combine HD maps and local maps such that the same planning modules can be used in both cases.

[Owen-Liuyuxuan]

Thanks for you quick response and I learned a lot about the current process.

I do not have experience using openpilot and I could not find any document about a map API. Can you explain your idea in more details ?

https://blog.comma.ai/0811release/#navigation . This official blog introduces the usage of map API from mapbox to get navigation results. To some extends this is like driving with Google Maps Directions. They will be updating much faster than OSMs and there will be real-time traffic information and more accurate speed-limit information. I expect we are able to handle this form of input.

[simon-eisenmann-driveblocks]

We will present the project next Thursday in a working group meeting (Planning & Control WG)!

[maxime-clem]

In the current version of Autoware the motion planner and controller expect inputs in map frame:

• /localization/kinematic_state with the pose of ego in map frame.
• /planning/scenario_planning/trajectory with trajectory points in map frame.

With a map-less architecture we need to assume that the map frame cannot be used for planning and control.
Therefore, I think we need to consider adding an odom frame (see https://www.ros.org/reps/rep-0105.html#odom) such that:

• Localization provides a pose in odom frame even when we cannot localize relative to a global map.
  • I believe this is already computed internally by the localization nodes and would be easy to add.
• Motion planning consumes a path in odom frame and produces a trajectory in odom frame.
• Control consumes a trajectory in odom frame.

I think for motion planning and control the change would be minor as they do not directly care about the map pose, but about the difference between the ego pose and the path/trajectory input, regardless of their reference frame.

If we do not want to add an odom frame to Autoware, we can consider a workaround where the map-less architecture would use the map frame for its output and would also publish the corresponding ego pose. The issue with this workaround is that it would prevent smoothly switching between a map-based and map-less architecture since it would cause inconsistent meanings of the map frame.

The following figures illustrate the two options.

odom + map frame	only map frame

[maxime-clem]

For a first prototype we are going to implement the "only map frame" approach.

@xmfcx's comments during the planning and control working gruop.

If we are doing localization on a global map then the odom frame can be equal to the odom frame.

I don't think there is any advantage in forcing odom = map. Meanwhile it will cause difficult transitions between driving on a global pointcloud/lanelet map (map frame) and driving with the map-less architecture (odom frame).

Concern about the quality of the odom frame.

I think these are our requirements for the odom frame:

• smooth (no sudden jumps).
• updated at the same rate as control (50Hz by default).
• error is small between updates of the Path in odom frame.
  • I don't know how to define "small", but I think it would be reasonable to have an error bellow 1cm between updates of the Path message.

I think the current odometry as used by the ekf_localizer meets these requirements but I need to confirm with the localization team.

Concern about how we use odom for global planning with the map-less architecture.

The odom frame is only meant to be used for short term planning and control. For global planning we can use GPS and some SD map (e.g., OpenStreetMap).

[lntsh854]

Hello, I would like to know what's the progress of this work so far?

[maxime-clem]

A prototype of the mission planner for local maps will be shared by the end of this month.

[maxime-clem]

A prototype implementation of the architecture proposed in this discussion has been made public here: https://github.com/tier4/mapless_autoware
It has been integrated with Autoware and tested to perform autonomous lane following and lane changes on a test track without using a pointcloud or vector map.

[lntsh854]

Very punctual release. Thank you

[Irvingzzzzz]

Hello! The Mapless architecture looks great, I hope you don't mind me asking, but I would like to understand the principles behind the implementation of RoadSegment and Mapless Planning. Are there any references or resources you could recommend?

[maxime-clem]

There is a description of the RoadSegments here: https://github.com/tier4/mapless_autoware/tree/main/mapless_architecture#mission-planner

The RoadSegments message contains multiple segments, each characterized by the following components:

• Linestrings: Two distinct linestrings that define the geometry of the segment.
• Neighbor IDs: Identifiers for neighboring segments.
• Successor IDs: Identifiers for successor segments.

Additionally, the RoadSegments message includes a pose attribute, which provides the position and orientation of the ego vehicle relative to the road segments. This structure allows for a comprehensive representation of road segments and their spatial relationships.

The Mapless Planning implementation is also summarized on that same documentation page. You can also check the README of each package to see an explanation of the module and its interface (inputs/outputs).

[jackistooloud]

A wonderful job! Thanks for generous sharing. But I'm still confused about some of the details and would like to get your reply.

I have one question about how the mapless_autoware code works with the original autoware. Since you provide an example launch file, mapless.launch.xml, for running Autoware along with the mapless architecture. But how to use the xml exactly? For example, I can run an autoware demo with the following command：

ros2 launch autoware_launch planning_simulator.launch.xml map_path:=$HOME/autoware_map/sample-map-planning vehicle_model:=sample_vehicle sensor_model:=sample_sensor_kit

So I'd like to know what the exact command for mapless_autoware is, and how the mapless folder and the original autoware folder are laid out.

Hoping for your reply!

[maxime-clem]

Thank you for your question.
I prepared the launch file but did not explain nicely how to use it. Since the file is not installed with a ROS 2 package, you need to run it "directly" with ros2 launch ./PATH/mapless_autoware/mapless.launch.xml map_path:=$HOME/autoware_map/sample-map-planning vehicle_model:=sample_vehicle sensor_model:=sample_sensor_kit (make sure to replace the PATH as needed).

But this launch file is mostly just an example and here are the important lines that were added:

# launch the local mission planner as described here: https://github.com/tier4/mapless_autoware/tree/main/mapless_architecture#launching-the-software
  <include file="$(find-pkg-share autoware_local_mission_planner)/launch/mission_planner_compose.launch.py"/>

# use the path calculated by the local mission planner as input to the behavior planner
  <node pkg="topic_tools" exec="relay" name="relay_path">
    <param name="input_topic" value="/mapless_architecture/mission_lane_converter/output/global_path"/>
    <param name="output_topic" value="/planning/scenario_planning/lane_driving/behavior_planning/path"/>
  </node>

[jackistooloud]

Thank you for such a quick reply, I'll try it.

[jackistooloud]

Well, I unzip the entire project and run the command in the current folder, but encountered the following error:

ValueError: './home/work/mapless_autoware/mapless.launch.xml' is not a valid package name

So do I need to vcs import build_depends.repos and compile the project with colcon build, like autoware?

[maxime-clem]

Yes the repository contains a number of packages that needs to be built and sourced.
You can either clone the repository in your Autoware workspace, or create a separate workspace where you will also vcs import src < build_depends.repos. In the second case, you will not be able to use the mapless.launch.xml file since it will try to launch packages from Autoware. Instead you can just launch the local mission planner with:

ros2 launch autoware_local_mission_planner mission_planner_compose.launch.py

[jackistooloud]

OK, I got it. Thanks for patient explanation.

[123sdadaw]

Thank you for your contributions to autoware's maplessness! I'm confused about the external odometry signal provider and hope to learn from your reply.
Since autoware does have a localization module, why to run the external odometer module instead of starting it directly in autoware at the same time as running Autoware along with the mapless architecture? After all, pose and velocity of the ego vehicle and acceleration data can be easily provided by IMU and odometry.

[maxime-clem]

Unfortunately the localization component of Autoware does not currently work without a pointcloud map.
You are right that the necessary data should already be available, so I believe minimal changes are required for the localization component to output an odometry signal without a map.

[123sdadaw]

Well, I'm trying to learn the wonderful project, but when I run mapless_autoware, I found that only centerline is visualized after providing external RoadSegment and /localization/kinematic_state, and other markers cannot be displayed, as shown in the following figure:

Plus, it seems that the vehicle did not move, which made me very confused.
I look forward to receiving your enthusiastic response!

[simon-eisenmann-driveblocks]

Hi! Thanks for your comment! We're planning to release an example bag soon to help test the mapless_autoware package. Please note that to drive the vehicle successfully, you'll also need a motion planner and a controller, as mapless_autoware only provides the goal path/trajectory.

[123sdadaw]

Thank you for your answer. Looking forward to the follow-up work!

[jackistooloud]

Hello! I noticed that mapless_autoware directly uses the centerline as a trajectory or path with no smoothing operations, whereas smoothing components in autoware such as path_optimizer work to require a map as well. Is there a subsequent consideration to add relevant optimizations?

[simon-eisenmann-driveblocks]

Hi! Currently, the motion planner is responsible for smoothing the trajectory/path, so we are not planning to add this functionality. However, this might change in the future.