The calculation of the control module takes time

[hzkkk1997]

I have tried to update the two visual plugins currently in the main branch to my v1.0 version and they work, but I found that the calculation time of the control module after stripping it out has increased significantly, to more than 150ms, this affects the actual control effect of the car. What might be the cause?

[maxime-clem]

Are you able to share your changes ? It is hard to know what could be the issue without knowing exactly what you changed.
But if you only changed things on the visualization side it should not impact the controller performance, so your issue is strange.

[hzkkk1997]

Thank you for your comment, but in fact, our independent control project is started in the same way as the original autoware.

[maxime-clem]

So you are using the ros2 launch autoware_launch autoware.launch.xml ... command but disabling all components excepts for control ?

[hzkkk1997]

Yeah, we did it the relatively simple way, trying to get through first.

[maxime-clem]

Can you share log files corresponding to each ros2 launch ... command you run in your system when the issue occurs ?
I will check if there is any useful message.

[hzkkk1997]

output.log
This is a separate log record after the start of the control project, the other modules are not convenient to upload at present, I will update as soon as possible in the future, sorry ~

[hzkkk1997]

@maxime-clem Hello, sorry to bother you. We seem to have solved the problem of which calculation takes time above by placing the individual control project in the same directory as other projects, but the reason is not clear at present.
https://github.com/orgs/autowarefoundation/discussions/5198#discussion-7159134
This question link would like to ask you for advice, what is a good way? Thanks a lot~

[hzkkk1997]

@maxime-clem Hi, sorry to interrupt again, but the situation seems to have only been resolved on one machine. So I wanted to consult you again, When the control part of the code is running, a timeout problem occurs. There are two Orin machines running the same control code, and the code execution times out on one machine, while the other runs normally. The control code on both machines is completely identical, and the related configuration libraries, such as the osqp library, are also the same. How to determine the cause of this problem？
The following figure shows the abnormal and normal calculation time of the control module.(blue line)


In abnormal circumstances, When operating the topic commands (topic echo, topic list), the time will also significantly increase.

[maxime-clem]

If just doing a ros2 topic list causes these delays, then the issue is probably caused by the communication cost over the network.

[hzkkk1997]

Thanks for your reply, I can understand that the communication pressure has increased, but I don't quite understand why the algorithm calculation time in the code has also increased.

[maxime-clem]

Without profiling it is hard to guarantee that it is the issue, but I can imagine that publishing or receiving messages (which happens during the "algorithm calculation") takes a lot of time. From what I understand, communication issues can cause any call using rclcpp (which includes publishers, subscribers, transforms, parameters, ...) to become laggy and cause performance issues.

[hzkkk1997]

Thank you very much for your answer！！ We will try to analyze it from the Angle you mentioned：）

[hzkkk1997]

@maxime-clem Hello, I'm very sorry to bother you. Recently, we used the tool "Flame graph" to try to analyze the calculation time of various functions in our project, and found that the time and times of various functions of eigen library called by the control project we separated were obviously different in different directories (/home/XXX/ directory and /mnt/ directory respectively). Do you know the possible reasons?
This is a situation where the time is significantly increased.

This is normal (under the /mnt/ directory).

[maxime-clem]

I know the Eigen functions become very slow when building with the DEBUG flag. Is it possible that one of your workspace was built with the DEBUG flag (instead of RELEASE) ?
Otherwise I don't know. You can try a clean build of the workspace to see if it fixes the performance issue (rm build install -rf before building).

[hzkkk1997]

Thank you very much for your suggestion. We'll try it right away.