A way to convert ros2 bags from one distribution to another

[chesterrc]

Description

A ros2 CLI tool to convert bag files to different distributions would be really helpful for collaborating with other users who have different ros2 distributions.

Related Issues

N/A

Completion Criteria

Argument in the CLI that would allow the conversion between different ROS2 distributions.

Implementation Notes / Suggestions

I would think that we want example metadata.yaml files of how different ROS2 distributions store their data as some sort of boilerplate.

[Jeffxzj]

Do you mean converting rosbag with different file formats? Like converting a SQLite3 formatted bag from older ROS2 distributions to an MCAP formatted bag using the ros2 bag cli ?

[lee-shun]

Do you mean converting rosbag with different file formats? Like converting a SQLite3 formatted bag from older ROS2 distributions to an MCAP formatted bag using the ros2 bag cli ?

Probably No. I think @chesterrc wants to ask if the bag recorded by Rolling can be played by Humble or vice versa. Because I got a similar request recently.

Do you have any suggestions?

[lee-shun]

Do you mean converting rosbag with different file formats? Like converting a SQLite3 formatted bag from older ROS2 distributions to an MCAP formatted bag using the ros2 bag cli ?

Probably No. I think @chesterrc wants to ask if the bag recorded by Rolling can be played by Humble or vice versa. Because I got a similar request recently.

Do you have any suggestions?

[MichaelOrlov]

Currently, the bag recorded by Humble can be replayed by all of its successor's distros up to the latest Rolling. We keep backward compatibility.
However, there is almost no way to make the bag recorded on Rolling to be replayed on Humble.
One workaround could be by deleting metadata.yaml file and run ros2 bag reindex path/to/robag2_folder on Humble.
The main difference is in the metadata serialized formats and internal data structures used when reading metadata.

One can see differences in a different version of metadata in unit tests https://github.com/ros2/rosbag2/blob/rolling/rosbag2_storage/test/rosbag2_storage/test_metadata_serialization.cpp
and in git history for the https://github.com/ros2/rosbag2/commits/rolling/rosbag2_storage/include/rosbag2_storage/bag_metadata.hpp

[lee-shun]

Oh，thank you for your quick reply！I will try this solution later！

Cheers 🍻

[jin3qin]

Currently, the bag recorded by Humble can be replayed by all of its successor's distros up to the latest Rolling. We keep backward compatibility. However, there is almost no way to make the bag recorded on Rolling to be replayed on Humble. One workaround could be by deleting metadata.yaml file and run ros2 bag reindex path/to/robag2_folder on Humble. The main difference is in the metadata serialized formats and internal data structures used when reading metadata.

One can see differences in a different version of metadata in unit tests https://github.com/ros2/rosbag2/blob/rolling/rosbag2_storage/test/rosbag2_storage/test_metadata_serialization.cpp and in git history for the https://github.com/ros2/rosbag2/commits/rolling/rosbag2_storage/include/rosbag2_storage/bag_metadata.hpp

Hi MichaelOrlov, after I followed your method, but it still doesn't work. This is the printing of the operation process.

qing@ubuntu:~/M300_160_1$ ls
M300 1601 0.mcap metadata.yaml metadata.yaml.cp
qing@ubuntu:~/M300_160_1$ rm metadata.yaml.cp
qing@ubuntu:~/M300_160_1$ rm metadata.yaml
qingdhp:~/M300_160_1$ ls
M300_160_1_0.mcap
qing@ubuntu:~/M300_160_1$ cd ..
qing@ubuntu:~$ ros2 bag reindex M300_160_1
[INFO][1738726137.875949969][rosbag2_cpp]: Beginning reindexing bag in directory:M300_160_1

closing.
[INFO][1738726137.911760334][rosbag2_cpp]: Reindexing complete.
qing@ubuntu:~$ ros2 bag play M300_160_1

closing.
[INFO][1738726147.948193008][rosbag2_player]: Set rate to 1
yaml-cpp: error at line 1,column 12: bad conversion

Now I have an idea, take the ros2bag of version Rolling and compile it under version Humble, and then try to play ros'bag, do you have any suggestions? Thanks!

[MichaelOrlov]

@jin3qin Can you share some example bag recorded on Rolling that you are trying to playback on Humble?

[jin3qin]

@MichaelOrlov Thank you for your reply, I can provide you with the Rolling recorded bag. How can I send it to you?

[MichaelOrlov]

@jin3qin The best way to share a bag file is to upload it somewhere with public access and put a link here in the thread.

[jin3qin]

@MichaelOrlov Hi, The recorded bag access link is as follows
https://drive.google.com/file/d/1j6uv-0lLX-t2_X6um3FPt_zQinvd6vYh/view?usp=drive_link

[MichaelOrlov]

@jin3qin Quick fix. try to manually edit metadata.yaml file by clearing offered_qos_profiles
i.e. try to make it as offered_qos_profiles: ""

[jin3qin]

@MichaelOrlov Thank you very much, I made it !

I delete the metadata.yaml file first, then run the ros2 bag reindex path/to/robag2_folder command to restore the file, and edit offered_qos_profiles: "" in the file. Finally, it play in Humble.

I express my gratitude once again.🍻🍻

[MichaelOrlov]

@jin3qin The reason why it was failing on offered_qos_profiles parsing is because we start using enum values instead of the integer values after the #1476.
It is not possible to fully backport #1476 becasue there are am API/ABI breaking changes. In particular here https://github.com/ros2/rosbag2/pull/1476/files#diff-dd3f71b4216dfd78804ed7ec306f84bc1c0564515dbb7bafb9c78ebc747b36b2

However, It could be possible to partially backport #1476 to support a new QoS format on a Humble. Needs to spend some effort and time on it.

[EmmanuelMess]

However, there is almost no way to make the bag recorded on Rolling to be replayed on Humble.

Why is this? Isn't the bag standardized? Or is it just the translation that is missing?

[MichaelOrlov]

However, there is almost no way to make the bag recorded on Rolling to be replayed on Humble.

Why is this? Isn't the bag standardized? Or is it just the translation that is missing?

The main reason is the absence of a backport of #1476 in the Humble.
In essence, we changed the format of how we serializing the offered_qos_profiles.
However, It could be possible to partially backport #1476 to support a new QoS format on Humble.
If someone could take this task it would be great.