[Proposal] vision-based localization with vector map

[KYabuuchi]

Hi! I have developed YabLoc, which is vision-based localization. Here is a demonstration video.
On behalf of TIER IV, I am pleased to announce a proposal to integrate YabLoc.

The yellow line in the upper left of the figure is the projection of Lanelet2 onto the image using the estimated self-position. They are not detected lines. If they match the actual road surface markings, it means that the position estimation performs well.

Currently, Autoware uses NDT for localization, mainly relying on LiDAR and point cloud maps. On the other hand, YabLoc does not require LiDAR and point cloud maps.
YabLoc extracts road surface markings from camera images and estimates the self-position by matching them with vector maps (Lanelet2). It estimates 3D position (x, y, z) and yaw angle, but not pitch and roll.

YabLoc will offer two initial pose estimation modes. One mode automatically estimates the orientation from the road markings, and the other uses RViz’s 2D Pose Estimate to specify the position.
Except for the auto initial pose estimation, YabLoc does not use DNN, and GPUs are not required.

Once YabLoc is merged to Autoware, Autoware users will be able to choose whether to use NDT or YabLoc (or eagleye) for localization. I believe this new localization method will enhance the functionality and versatility of Autoware.

Method

Input & Output
LiDAR and PCD map are no longer needed in YabLoc and camera images are required.
Also, it is important that the Lanelet2 includes road surface markings.

The basic principle of YabLoc
YabLoc uses a particle filter. The basic principle is illustrated in the diagram below. It extract road markings by masking the line segments by the road area obtained from graph-based segmentation. The red line in the center-top of the diagram represents the line segments identified as road surface markings. YabLoc then transform these segments for each particle and determine the particle's weight by comparing them with the cost map generated from lanelet2.

Localization node diagram when YabLoc is merged

The following diagram shows the node diagram. The blue blocks represent existing nodes in Autoware, while the yellow blocks represent nodes in YabLoc. YabLoc acts as a pose estimator instead of NDT.

(Some nodes are omitted for clarity.)

Limitation

• YabLoc does not estimate roll and pitch, therefore some of the perception nodes may not work well.
• I have tested YabLoc with the front and rear camera for now. (It will probably work with other installation ways.)
• It doesn't support multiple cameras now. But it will in the future.
• It strongly depends on GNSS, IMU and odometry in places where there are few road surface markings, such as intersections.
• Since Yabloc matches with Lanelet2 to localize, if the road boundary or road surface markings are not included in the Lanelet2, the estimation fails.

Evaluation Environment

I tested YabLoc for some dataset in TIER IV.

Although I cannot show images or videos due to some reasons, I also verified autonomous driving at low speeds in closed test coarse.

• freeway

• tunnel

• urban road

• AWSIM

Autoware Integration Plan

I plan to merge as follows referring to Eagleye:

• Place YabLoc in universe/external by adding autoware.repose in autoware.
• Add an option to pose_initializer to call the initial position estimation of YabLoc.
• Add a launch file to run YabLoc.

I plan to create a merge request to autoware.universe, autoware_launch, autoware soon.

In the future, if it becomes difficult to maintain YabLoc in a separated repository or becomes commonly used, I plan to place YabLoc directly in autoware.universe.

Please let me know if you have an opinion about how to integrate 🙏

Current Status

• YabLoc is now available to the public. https://github.com/tier4/YabLoc
  • You can run the demonstration with a sample rosbag. (It does not depend on Autoware.)
• I am creating a YabLoc integration branch in my fork repository.
  • GitHub - KYabuuchi/autoware.universe at feature/yabloc_integration
  • GitHub - KYabuuchi/autoware_launch at feature/yabloc_integration

---

Thank you for your support, and I look forward to your feedback!

[n-patiphon]

Awesome work!

I see you use the center line of road markings from the HD map for the association (matching). I wonder if it would be helpful for your algorithm if you also knew the width of those road markings from the HD map.

In Lanelet2, I think road markings such as solid lines and dashed lines are normally represented using the center spline without width, so you might not be able to get the width information. Although line types can be specified using tags, e.g., thin_solid, I'm not sure if these tags have specified widths in meters (or centimeters).

On the hand, I see that the crosswalks in the map you used contain locations of "stripes". Do you think it would work well if crosswalks were only defined using their boundaries (the outline of the crosswalks without strips inside)?

[KYabuuchi]

Thanks for your interest!

I see you use the center line of road markings from the HD map for the association (matching). I wonder if it would be helpful for your algorithm if you also knew the width of those road markings from the HD map.

Currently YabLoc treats all road markings as line segments and not as regions. It is convenient to treat all markings as line segments because it allows for a unified processing for road surface markings that have thickness and those that do not.

The width of the road surface markings is not used now, but it would be a good idea to use it to generate cost map images. The cost map image is currently generated as shown in the figure below.

There should be width at the peaks in the cost map image, but there isn't now. If cost maps properly account for width, accuracy may improve.
Thanks for your valuable opinion. 👍

---

On the hand, I see that the crosswalks in the map you used contain locations of "stripes". Do you think it would work well if crosswalks were only defined using their boundaries (the outline of the crosswalks without strips inside)?

Probably YabLoc does not work well with only boundaries at intersections because it does incorrect matching.
I will add the feature that YabLoc does match incorrectly if polygons with tag such as 'unmapped_area' are embedded in lanelet2. 👍

[n-patiphon]

Thank you for your reply! Looking forward to the new features!

[VRichardJP]

Very impressive!

Once YabLoc is merged to Autoware, Autoware users will be able to choose whether to use NDT or YabLoc (or eagleye) for localization.

Why not all at the same time? EKF localization gets better as it receives more and more diverse data, no?

[KYabuuchi]

Thank you for your interest!

The reason for not launching multiple methods at the same time is because YabLoc is computationally heavy.
In addition, because of the superior NDT, there would be little advantage to launching YabLoc simultaneously.

Once YabLoc is merged to Autoware, Autoware users will be able to choose whether to use NDT or YabLoc (or eagleye) for localization.

Note that this is the default configuration, and if you edit the launch file appropriately, it will be possible to connect multiple pose_estiamtor to EKF at the same time.

[KYabuuchi]

I also plan to implement runtime switching between NDT and YabLoc.
Once that is implemented, Autoware would be able to localize with YabLoc where there are no PCD maps and localize with NDT where there are PCD maps.

[VRichardJP]

That looks like what EKF can do with some extra steps ^^

if you edit the launch file appropriately, it will be possible to connect multiple pose_estiamtor to EKF at the same time.

As far as I know, EKF only supports 1 pose topic. It is possible to relay multiple pose topics to the same EKF pose input, but then all pose use the same parameters (e.g. pose_smoothing_steps). Or do you mean someone is working to make EKF accept many inputs at the same time?

[KYabuuchi]

I mean that both NDT and YabLoc output pose_estimator/pose_with_covariance simultaneously and one callback in EKF handles them.
I don't think using the same parameters related to the pose measurement would be a problem. 🤔

[KYabuuchi]

Here's the latest update. 📣
YabLoc has been merged into autoware.universe!

• feat(yabloc): add camera and vector map localization autoware.universe#3946
• feat(yabloc): add camera and vector map localization autoware_launch#393

If you're interested in running autoware with YabLoc mode, I recommend trying the sample dataset provided in the "Tests performed" section of this PR

[KYabuuchi]

I am working on updating the Documentation page.

[liuXinGangChina]

hi @KYabuuchi yabuuchi-san，really great work
I notice that you only use the marks on the ground，do you consider using vertical features like street lamp or pillars to enhance longitudinal localization. I saw a implementation in this paper

[KYabuuchi]

@liuXinGangChina Thank you for your interest.
As you mentioned, this method currently relies solely on ground markings. Unfortunately, I do not have plans to add functionality that utilizes vertical features.
My approach involves matching projected images on the ground, so transitioning to using vertical landmarks would require significant changes.