Split and version control of autoware_common repository

[youtalk]

We propose to separate the autoware_common repository into separate repositories based on package type. It allows each package to be versioned independently.
It will also allow us to release packages from the ROS build farm using bloom, which will reduce build time and increase awareness from the ROS community.

The new repository configuration is envisioned as follows:

• autoware_cmake
  • autoware_cmake
  • autoware_lint_common
• autoware_utils
  • autoware_utils
• autoware_lanelet2_extension
  • lanelet2_extension
  • lanelet2_extension_python

At the same time as the package separation, we will also work on adding the autoware_ prefix for each package and namespace mentioned by @esteve.

https://github.com/orgs/autowarefoundation/discussions/4097
#4569

Considering the transition effort, we plan to leave the autoware_common repository as is for the time being until the package releases from the ROS build farm will be completed.

This time, autoware_common is our target, but we plan to gradually expand the scope to other packages on the Autoware repositories.

[esteve]

This is a good idea, it'd let us tag and release each autoware_common related packages independently from the rest of Autoware, though I think those packages would be of little value outside Autoware.

Regarding releasing packages into to the ROS buildfarm via bloom, that's something I've been working on for the past several months, so please reach out to me, it'd be good to coordinate efforts. In the meantime until we properly tag and release packages, I've set up https://github.com/autowarefoundation/autoware-deb-packages which contains all of Autoware as Debian packages for ROS 2 Humble and Ubuntu Jammy. This is an intermediary step towards properly releasing these packages into the ROS buildfarm, but we couldn't do that because of the lack of a release process.

@xmfcx @mitsudome-r I see that https://github.com/orgs/autowarefoundation/discussions/4668 was already created and closed days ago and that the https://github.com/autowarefoundation/autoware_lanelet2_extension repository was created without much discussion, shouldn't this have been brought up in the ASWG meeting before creating these repositories?

[youtalk]

I'm still getting used to the development progress under the Autoware Foundation since I'm new to TIER IV. I apologize for creating a new repository without discussion.
We will be sure to keep you informed as we proceed with the package release.

[esteve]

No problem 🙂 Thanks for creating this discussion, it'd be great to have Autoware included in the ROS buildfarm!

[doganulus]

Before going that way, it is essential to check the limitations of bloom. I will not go deep, but it is not hard to see that bloom is another half-baked tool from the ROS ecosystem.

For example, currently, Bloom does not support generating runtime and development debs from packages:
https://answers.ros.org/question/411766/use-bloom-to-generate-a-deb-and-a-devdeb-packages/

If you are interested, there is a related PR where you can review to make it go faster: ros-infrastructure/bloom#703

But, of course, it hits the maintenance and human resources problem of the ROS ecosystem. Therefore, I think it is better for the Autoware project to lead the ROS ecosystem with the use of modern industry-standard tools rather than legacy solutions. This is a progress.

[mitsudome-r]

@doganulus Thanks for the comment. Although I agree that there are limitation with ROS boom, I think we can first focus on generating distributing debian package with the current infrastructure. At the moment, we don't even have the binary distributed in ROS community, and this should work lower the barrier for ROS users to enter Autoware community as well. Then, we can update the infrastructure to deliver more sophisticated way.

[doganulus]

I like @esteve's work using GitHub Actions here. It's a modern way to generate Debian packages. What I am saying is Bloom would be a regression over his work.

Also, lowering barriers for ROS users means raising barriers for everyone else. What is the number of developers who use GHA, and what is the number for Bloom? Such moves would make the project target an even smaller, burnt-out user base. You may not see it as you are too familiar with the ecosystem. But let me say this: Autoware as a ROS project is uninteresting. Autoware, as a software-defined vehicle project, however, would get many new enthusiasts and support. This difference is critical for success.

[esteve]

FYI, the GitHub action we use for generating Debian packages uses bloom under the hood (https://github.com/esteve/ros-deb-builder-action/blob/tier4/main/build#L106)

[xmfcx]

Over all, I am alright with splitting into multiple repositories. But I need to make some comments on the rest.

cc. @esteve @youtalk @mitsudome-r

Delays from depending on the debian packages for CI

My main concern is about the bloom release delays.

Sometimes, we need to make changes in multiple repositories.
Example:

• a change in autoware.universe
• a change in autoware_common (autoware_utils package is a common example)

And the autoware.universe change depends on the autoware_common change.

The usual method is to first apply autoware_common change.
Then apply autoware.universe change.
And CI will use the updated autoware_common since we are rebuilding from scratch.

But if we offload autoware_utils to the bloom and autoware.universe depends on the .deb package from bloom, there will be a huge delay.

I don't know how large it will be but I know it can take weeks.

Example:
Today is: 2024-05-02
Last update in http://packages.ros.org/ros2/ubuntu/dists/jammy/ is: 2024-04-30

I've looked into it a bit and found out about: https://docs.ros.org/en/humble/Installation/Testing.html for nightly releases.
Additional links:

• https://ci.ros2.org/view/nightly/
• https://build.ros2.org/job/Hci__nightly-release_ubuntu_jammy_amd64/

I don't know how these interact with each other and how the schedules are.

But the best case scenario is that we will have to wait for a nightly build to finish before merging the dependent PR.

Autoware only packages

• autoware_cmake
• autoware_lint_common

These packages don't consume build time since they don't have anything to build. We won't save build time if we use them from bloom.

[doganulus]

I see these those changes are already getting implemented: https://github.com/ros2-gbp/?q=autoware&type=all&language=&sort=

Sorry, but there is no discussion here.

[esteve]

Yes, there was a discussion, and we agreed to release these packages to the ROS buildfarm

[doganulus]

This is not a Tier IV-only project. Please use your own fork to try and go fast with such implementations if you think it is worth it.

Unfortunately, these issues are getting more frequent and making me worried.

[esteve]

The discussion about releasing packages into the ROS buildfarm has happened in the open here for over a year #3360 and https://github.com/orgs/autowarefoundation/discussions/4097

[doganulus]

And I see no agreement in place in those threads. Actually, it is the opposite. For example, @mitsudome-r says in the thread:

As Core is currently empty, we should start with porting Universe packages into Core first before doing this task.

This has not happened as far as I know. Autoware has problems with software quality, and somehow, you decided to release them this week. Therefore, please first demonstrate you can build runtime and devel packages for components on TierIV's repos before releasing it anywhere. Then, you can move them into the Autoware organization. I keep my Autoware container repos separately for example.

[youtalk]

The repository split is complete and releases from the ROS build farm of the core packages are well ongoing.
Thank you for all the discussion.

• https://github.com/ros2-gbp/autoware_cmake-release
• https://github.com/ros2-gbp/autoware_utils-release
• https://github.com/ros2-gbp/autoware_lanelet2_extension-release
• https://github.com/ros2-gbp/autoware_adapi_msgs-release
• https://github.com/ros2-gbp/autoware_msgs-release
• Update autoware.tf to create autoware_internal_msgs-release ros2-gbp/ros2-gbp-github-org#516
• Add autoware_cmake to humble, jazzy, and rolling indices ros/rosdistro#41039
• Add autoware_utils to humble, jazzy, and rolling indices ros/rosdistro#41040
• Add autoware_lanelet2_extension to humble, jazzy, and rolling indices ros/rosdistro#41124
• Add autoware_adapi_msgs to humble, jazzy, and rolling indices ros/rosdistro#41029
• Add autoware_msgs to humble, jazzy, and rolling indices ros/rosdistro#41122
• Add autoware_internal_msgs to humble, jazzy, and rolling indices ros/rosdistro#41123

[youtalk]

It should be closed when the Dockerfile and .repos migration is complete. That is why I've reopened.

[youtalk]

All packages were released from ROS buildfarm and the version in autoware.repos is now specified with tags instead of the main branches #4963.

Thank you all for discussing about it.

[esteve]

Thanks for all the work! 🙂