Setting qos_overriding_options in create_publisher or create_subscription for a new Pub or Sub from within a dynamic parameter callback creates a deadlock condition when resource locking is required

[SteveMacenski]

Operating System:

Ubuntu 24.04

ROS version or commit hash:

Rolling

Client library (if applicable):

rclcpp

Steps to reproduce issue

We use create_publisher for creating publishers generally and I have been working to expose more of the advanced features within Nav2's LifecycleNode. Including among them qos_overriding_options for overriding the QoS policies based on a single parameter allow_parameter_qos_overrides.

I found that generally this works fine and there is no issue in the lifecycle transition functions. However, when using create_publisher in the callback of a dynamic parameter to reinitialize an object, the program deadlocks at rclcpp::create_publisher.

What's interesting is that in all cases, these publishers previously existed so the parameters should have already been declared that this logic is declare or getting. That makes me think that (a) its declaring again and that causes a problem or (b) the get logic is wrong and this is being uncovered by running this 2x.

I suspect this is also true of create_subscription, but create_publisher is the only one I was debugging at the moment. I see some tests passing in my refactor with subscriptions interestingly enough, but I'm not sure if that is because we didn't test them thoroughly enough or it really is only create_publisher that has this error

Expected behavior

Publisher is created and program can continue as normal

Actual behavior

Publisher is created and program is permanently deadlocked

Additional information

No response

[SteveMacenski]

This is an example application usage: https://github.com/ros-navigation/navigation2/blob/main/nav2_smac_planner/src/smac_planner_hybrid.cpp where as we create a new costmap publisher on configure lifecycle state and again in the dynamic parameter callback if required. If the qos overriding options is non-empty, then it deadlocks.

[SteveMacenski]

Related:

• Cannot create subscription/publisher with QoS override parameter via parameter callback. #2741
• enable parameter update recursively only when QoS override parameters. #2742

But I can still see this with the most recent binaries of rclcpp 29.6.0 for publishers. Interestingly enough ros-navigation/navigation2#4968 works now which was the Nav2 PR that brought up the tickets above, but that uses a subscription.

[fujitatomoya]

@SteveMacenski thanks for the issue description.

#2742 adds the test for both publisher and subscription including recreating the objects, and it is passing. for more details about the deadlock, can you make the self-contained reproducible example test for this issue? i want to know which mutex lock (if that is actually the deadlock) cannot be acquired. and why subscription does not have this problem so far??? that i have no idea at this moment...

[SteveMacenski]

I added some logging and it was the mutex because of the deadlock. Basically:

• the mutex was locked on Call1 -> new interface was made and hung at create_publisher
• Then Call2 came in -> deadlocked because it could not obtain the lock, but it the program was already hung at create_publisher which created the issue

Call2 I think is due to the creation of the publisher, triggering another parameter callback which causes the deadlock on accessing the mutex. I can't reproduce with a subscription.

The hang happens from this path when any of the QoS overrides are enabled

• https://github.com/ros-navigation/navigation2/blob/main/nav2_smac_planner/src/smac_planner_hybrid.cpp#L836 (has the mutex lock)
• https://github.com/ros-navigation/navigation2/blob/main/nav2_smac_planner/src/costmap_downsampler.cpp#L54 (node.expired() is false)
• https://github.com/ros-navigation/navigation2/blob/main/nav2_costmap_2d/src/costmap_2d_publisher.cpp#L71-L80
• https://github.com/ros-navigation/navigation2/blob/main/nav2_ros_common/include/nav2_ros_common/lifecycle_node.hpp#L148
• https://github.com/ros-navigation/navigation2/blob/main/nav2_ros_common/include/nav2_ros_common/interface_factories.hpp#L170
• Note: I checked by breaking out createPublisherOptions and logging around declare_or_get_parameter, neither of those are the issue. It hangs inside of rclcpp::create_publisher

I see a second call that then deadlocks trying to obtain the mutex (https://github.com/ros-navigation/navigation2/blob/main/nav2_smac_planner/src/smac_planner_hybrid.cpp#L615) that the main hung call is hogging.

I made a reproduction example and can't seem to reproduce it with https://github.com/open-navigation/opennav_rclcpp_issue_reproduction. I do see however if I remove those publishers, then everything works properly.It seems pretty cut and dry from the logging that it hangs creating the publisher triggers another parameter update for some reason that causes the deadlock. I also note that this is the only publisher I can find in Nav2 that is recreated in a dynamic parameter callback, so it makes sense why I only see it in one test. We have several subscriptions that are recreated which are working fine.

I just can't figure out how to apparently reproduce without the chain of events in the Nav2 path I list above...

Edit: I haven't been able to reproduce yet outside of the smac planner, but I have been able to show it in the smac planner with the callback containing nothing other than the lock and creating a generic publisher that could be to any topic / type.

rcl_interfaces::msg::SetParametersResult
SmacPlannerHybrid::dynamicParametersCallback(std::vector<rclcpp::Parameter> parameters)
{
  rcl_interfaces::msg::SetParametersResult result;
  std::lock_guard<std::mutex> lock_reinit(_mutex);
  (void)parameters;

  auto node = _node.lock();
  auto costmap_pub_ = node->create_publisher<nav_msgs::msg::OccupancyGrid>(
    "h",
    nav2::qos::LatchedPublisherQoS());

  result.successful = true;
  return result;
}

[SteveMacenski]

OK. I found root cause @fujitatomoya

The sub / pubs work fine for creating the interfaces again in the dynamic callbacks as long as those interfaces were already existing in the node in the first place. If they're new, and therefore need to declare, that's where the hiccup is.

In the Smac Planner, we only create the object that is hanging on us if the initial settings say so (example), so when we create it in the dynamic configuration callback, that is the first time it is ever created which causes the hang because its declaring the parameter (I assume). When I make this object always created (as in here) then it works fine.

I was able to show this by adding an arbitrary publisher into the dynamic parameters callback of any node and see the same behavior. I also shows this happens for subscriptions as well.

So, the issue is that when a Pub/Sub that enables parameter overrides is created for the first time in a dynamic parameter callback, it hangs when the resource needs to be locked because it triggers a call to the dynamic parameters callback about the declaration.

[fujitatomoya]

@SteveMacenski thanks for the information.

1st thing as work-around for your application, the resource needs to be locked can use recursive mutex lock? this still makes sure that the resource is locked during multiple parameter callback chain. or i would check the what parameter is being changed in the callback 1st, and then lock the resource if necessary.

i believe that the above is work-around and that is actually a burden for the user application.
the question that i have is that, do we really need to call the user callback when qos override parameters are being declared? probably ordinary parameter requests to those qos override parameters need to be monitored by the user callback, because user callback would watch those qos setting in the callback to accept or reject the incoming requests. but i am not sure if we need to call the user callback on declaration when it creates the publisher and subscription endpoints. this could avoid the callback chain issue reported here?

[SteveMacenski]

do we really need to call the user callback when qos override parameters are being declared?

I think not. I think those are internal details.

What I ended up doing was to slightly adjust how we approach the initialization so that the publishers are created before the dynamic parameter callback can be triggered. That resolved it, but the recursive mutex would as well. This report isn't blocking Nav2 anymore, but still a good topic for fixing since that could cause some users issues especially if one of those dynamic parameters is something like verbose or debugging whereas several publishers may be initialized for use.