Adding Multidrone Deployment Feature with Zenoh-ROS2 by N0OBSTUDENT · Pull Request #74 · learnsyslab/lsy_drone_racing

N0OBSTUDENT · 2026-03-27T22:25:40Z

Multi-drone real race support via host-client ROS2 architecture

Host-client split: Replaces the monolithic multi_deploy.py with separate deploy_host.py
and deploy_client.py scripts. The host manages all Crazyflie workers and race coordination;
each client runs on its own compute unit and controls a single drone.
New environment classes: Adds RealMultiDroneRaceEnvHost and RealMultiDroneRaceEnvClient —
Gymnasium-compatible environments for the two sides of the host-client protocol.
ROS2 race communication layer: New RaceCommNode in ros_race_comm.py handles all
inter-process messaging (host-ready, race-start, client-state, clock calibration) using
lsy_race_msgs ROS2 message types.
Clock synchronisation: Clients calibrate their clock offset against the host via a ROS2
service before the race starts.
Multi-drone controller wrappers: Adds AttitudeController and AttitudeMPC subclasses
that slice the per-drone observation from the joint multi-drone obs dict using the drone's rank.
Unknown observations default to NaN: Considering that other drones could be out of mocap range, set the default observation to np.nan
Config updates: multi_level2.toml and multi_level0.toml updated to fit the new
architecture.

* Add pngs to installed files * Bump workflow pixi version * Fix cam config * Update lock file

…earnsyslab#65) - Fix UnboundErrors raised in finally blocks if creating a ros connector fails - Set SCIPY_ARRAY_API flags in scripts

-barrier is shared via socket -barrier setup in configuration -barrier could start by any deploy node, or standalone barrier process started by host

…lti-drone racing configuration Other dimension problem fixed within deploy

…_drone_racing into zenoh_multidrone

…t processes

…titude rl

N0OBSTUDENT · 2026-04-02T13:59:42Z

And here is the test cases I would like to do, but after we made the structure clear.
https://docs.google.com/spreadsheets/d/1a_WvakzvkbI9FE5ZeVwlMI_acAK3nL8hTaW2EinpsPs/edit?usp=sharing

ratheron

Generally nice implementation with proper safe guards. Thank you for the work!

ratheron · 2026-04-03T10:22:19Z

Please make sure the initial positions etc are equal between the levels

ratheron · 2026-04-03T10:25:50Z

Why is this necessary for this PR?

ratheron · 2026-04-04T13:36:43Z

+        self.taken_off = False
+
+
+class RealMultiDroneRaceEnvClient(Env):


Why not inherit from RealRaceCoreEnv? Much of the functionality should be identical.

ratheron · 2026-04-04T13:58:17Z

+logger = logging.getLogger(__name__)
+
+
+class CrazyflieWorker:


This class should also inherit RealRaceCoreEnv. Or better: This class is used by the real race core env. In that case, we should move it somewhere else.

amacati

Left some comments on the surrounding changes.

TODO @amacati review ros_race_comm, real_race_env_client, real_race_host.

amacati · 2026-04-05T10:52:42Z

General comment on the ros stuff: I am not happy at all with having a ros_ws in a python package. Ros packages should go into whatever catkin workspace is already there on the system, not be included inside a Python package. How do other packages handle this (apart from crazyswarm)? Do we need these messages? Can't we build them out of existing ones?

Now I am cloning the message from an external repository, following the style of mocap pacakge. We will be sharing the pacakge with simulation on https://github.com/rducrist/drone_racing_msgs

amacati · 2026-04-05T11:00:49Z

+    except Exception as e:
+        logger.error(f"Host encountered error: {e}", exc_info=True)


Why do we need a bare except here? That's an absolute last resort. Finally triggers anyways

The problem now is that multiple processes may throw tons of messages. I was trying to make the belonging of the message clearer

amacati · 2026-04-05T11:03:24Z

+
+
+def main(
+    config: str = "multi_level2.toml", controller: str | None = None, drone_rank: int | None = None


How do we ensure the same configs are selected on all clients? Do the settings need to match somehow, or are heterogeneous settings even desirable?

Currently there is no way we could ensure that... I suppose the setting do not need to match perfectly for every client. Only the track and drone settings need to be the same

amacati · 2026-04-05T11:04:27Z

+    if drone_rank is None:
+        raise ValueError("drone_rank must be specified")


None does not make sense in the first place. Why type hint a None if None is not possible? Why default to a value that throws an error?

amacati · 2026-04-05T11:09:52Z

+
+    Observation space:
+        A dictionary containing the state of all drones in the race, mirroring
+        :class:`lsy_drone_racing.envs.multi_drone_race.MultiDroneRaceEnv`.


That's the sim class, and it should stay

amacati · 2026-04-07T12:20:54Z

+        if self._comm is None:
+            self._init_comm()
+
+        current_pos, _, _, _ = self._get_all_drone_states()


Nit: Avoid getter names

amacati · 2026-04-07T12:32:43Z

+                self._send_state_update(
+                    np.zeros(4 if self.control_mode == "attitude" else 13), stopped=True
+                )
+                time.sleep(0.1)  # allow the executor thread to flush the message before shutdown


Can we ensure this instead of just waiting? I.e. send something with flush=True?

I will test whether the problem still exists rightnow. By then I was encountering issues with ros publisher closing too fast before the last message is sent

amacati · 2026-04-07T13:25:53Z

+    @property
+    def node(self) -> Node:
+        """The underlying rclpy node."""
+        return self._node


Why do we need the property then?

That's my mistake. Changed it in N0OBSTUDENT@2ed0147

amacati · 2026-04-07T13:48:32Z

+logger = logging.getLogger(__name__)
+
+
+class CrazyflieWorker:


Could we turn this into one function instead? We only execute run() anyways

It is doable, but the function will be as long as a tiolet paper roll

amacati · 2026-04-07T13:49:38Z

+class RealRaceHost:
+    """Base class for multi-drone race hosts.
+
+    Subclasses implement :meth:`load_config`, :meth:`connect_drones`,
+    :meth:`host_main_loop`, and :meth:`close` for a specific drone platform.
+    """


Why do we subclass this? We're only running crazyflies for now. Also, if anything, this should then be a protocol

My initial is that we could have different implementation of the Host with common communiation between the client and the host. So I had communicators initialized in RealRaceHost...

amacati · 2026-04-07T13:50:33Z

+def _suppress_shutdown_thread_errors():
+    """Install a threading.excepthook that silences expected ROS2 shutdown exceptions.
+
+    Replaces the noisy default traceback for :class:`~rclpy.executors.ExternalShutdownException`
+    and :class:`KeyboardInterrupt` in background spin threads with a single DEBUG log line.
+    Any other uncaught thread exception still goes through the default handler.
+    """
+    _original = threading.excepthook
+
+    def _hook(args: threading.ExceptHookArgs) -> None:
+        if args.exc_type in (ExternalShutdownException, KeyboardInterrupt) or (
+            args.exc_type.__name__ == "RCLError"
+        ):
+            logger.debug(f"Thread '{args.thread.name}' stopped (shutdown)")
+        else:
+            _original(args)
+
+    threading.excepthook = _hook


This is fairly complex. Was this a major issue during testing on the real hardware, or why was this included?

And use only one stop event for now

Now the setup_mocap.sh script clones an external repository including the message at https://github.com/rducrist/drone_racing_msgs

Co-authored-by: Copilot <copilot@github.com>

ratheron and others added 22 commits December 20, 2025 00:01

Fix cam config and add pngs (learnsyslab#62)

c6daa85

* Add pngs to installed files * Bump workflow pixi version * Fix cam config * Update lock file

Fix UnboundError when trying to close an uninitialized ROSConnector (l…

fe35a54

…earnsyslab#65) - Fix UnboundErrors raised in finally blocks if creating a ros connector fails - Set SCIPY_ARRAY_API flags in scripts

Merge remote-tracking branch 'origin/main' into dev

ae6fedd

Protptype for multi drone racing using a Barrier object to sync start

674f981

-barrier is shared via socket -barrier setup in configuration -barrier could start by any deploy node, or standalone barrier process started by host

Fixed dimension problem when trying to intialize a controller with mu…

21d7b1e

…lti-drone racing configuration Other dimension problem fixed within deploy

Vibe coded 1st version of multi drone racing using Zenoh

2d59f68

Added real race client

74e8d27

Fixed all bugs to make it run

5079284

Added latency calibration

5764cad

Make unknown observation nan instead of zero

f12bbf1

Merge branch 'zenoh_multidrone' of https://github.com/N0OBSTUDENT/lsy…

6f4c80d

…_drone_racing into zenoh_multidrone

Switched to time.time() from perf_counter()

1e373f6

Made it work

756c5aa

Added rank to other controllers

b87f41a

Made a demo for different controller, different frequency in differen…

df93e96

…t processes

Clean up codes

d7f6654

Removed raw-zenoh and takeoff barrier related code

22f5aa3

Edit host, client and comm node code to make it look neater

7d30658

Resolve naming issues

4355b63

Split the multi and single controllers

c869263

Pass ruff checks

b4b790e

Added ROS2 package

03222bc

N0OBSTUDENT requested review from amacati, Copilot, ratheron and yufei4hua March 27, 2026 22:25

N0OBSTUDENT assigned amacati, ratheron, N0OBSTUDENT and yufei4hua Mar 27, 2026

N0OBSTUDENT added 3 commits March 28, 2026 22:40

Organize imports, docstring and output format to pass ruff check

46212bc

Restored attitude rl to latest commit in dev and introduce wrapped at…

86dfcbf

…titude rl

Pass ruff check

fec0381

N0OBSTUDENT marked this pull request as draft March 29, 2026 14:26

ratheron added the enhancement New feature or request label Apr 3, 2026

ratheron requested changes Apr 4, 2026

View reviewed changes

amacati reviewed Apr 5, 2026

View reviewed changes

amacati reviewed Apr 7, 2026

View reviewed changes

ratheron mentioned this pull request Apr 11, 2026

Enable n controllers to run in multisim #77

Merged

ratheron force-pushed the dev branch 2 times, most recently from 51e8bc2 to 4a63d77 Compare April 13, 2026 22:00

N0OBSTUDENT and others added 18 commits April 21, 2026 03:33

Replace events with barrier.

fc5fa94

And use only one stop event for now

keep only one example for multi-drone racing wrapping

767a622

Modified ros2 messages.

0940e1c

Now the setup_mocap.sh script clones an external repository including the message at https://github.com/rducrist/drone_racing_msgs

Made a temporary repository to test the message

4a810db

Reused the EnvData from real_race_env.py

78c2965

Merge the ROS connectors

6fdd078

Fixed wrong docstring

c895bee

Co-authored-by: Copilot <copilot@github.com>

Changed drone_rank check to assert, removed redundant deploy variable

eccd03c

docstring in attitude_controller_multi.py is shorter now

9142d42

Fixed one ignored naming change

287dab3

Avoided the getter function naming

a3f5b94

Spreaded out the safety limit bound condition

bf96661

Removed the 0.1sec waiting and bare exception at shutdown of client

a8dcf25

Split the ConfigDict argument in RealRaceHost

20919df

Fixed bad docstring

b671a74

Replace the busy-wait loop with event blocking the thread

bbd9c31

Remove the property and make the node externally accessible

2ed0147

Remove unnecessary imports

2458860

		self.taken_off = False


		class RealMultiDroneRaceEnvClient(Env):

		except Exception as e:
		logger.error(f"Host encountered error: {e}", exc_info=True)



		def main(
		config: str = "multi_level2.toml", controller: str \| None = None, drone_rank: int \| None = None

		if drone_rank is None:
		raise ValueError("drone_rank must be specified")

Conversation

N0OBSTUDENT commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Multi-drone real race support via host-client ROS2 architecture

Uh oh!

N0OBSTUDENT commented Apr 2, 2026

Uh oh!

ratheron left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

amacati left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

N0OBSTUDENT commented Mar 27, 2026 •

edited

Loading