Stable flight with direct RPM control in Crazyflie 2.X?

[jediPhD]

I'm trying to train a PPO agent to control a single Crazyflie 2.X in gym-pybullet-drones (Physics.PYB). My action space is direct motor RPMs (4 motors), normalized to [-1,1] and then scaled to [0, max_rpm]. However, the drone becomes highly unstable immediately after takeoff: roll/pitch angles exceed 30° within a few steps and it crashes. The angular velocity seems to saturate at ~115°/s, but the tilt keeps increasing.

I have tried:

• Increasing PID gains (kp_roll/pitch up to 40, kd up to 15)
• Clipping the action to smaller ranges (e.g., [-0.5,0.5])
• Using Physics.DYN instead of PYB
• Reducing control frequency (from 48 Hz to 24 Hz)
• Adding a safety layer that reduces action when tilt > 10°

None of these prevented the crash. The only thing that works is to use a high-level controller (e.g., output position deltas and let a PID convert them to RPMs), but I would like to understand if direct RPM control is inherently unstable in this simulator.

My questions:

1. Is it physically/algorithmically possible to achieve stable flight with direct RPM control using RL?
2. If yes, what are the key tricks (action scaling, reward shaping, simulation parameters) that make it work?
3. If not, is there a known reason (e.g., the physics engine's angular velocity clamping) that makes this approach infeasible?

Any insights or references would be greatly appreciated!

[visionofdavinci]

Hello, I am still a student but I have been working on PPO-based drone control for my thesis using this simulator so I've run into a lot of the same issues. Here are some ideas about what I've learned related to your questions that could maybe help:

Question 1: Is it physically/algorithmically possible to achieve stable flight with direct RPM control using RL?
Answer:

• To answer shortly yes, but it is way harder than using a high-level controller. The difficulty of this question is coming from the fact that you are asking the policy to learn both low-level stabilization and task-level behaviour since you are using direct RPM control, which creates a much harder credit assignment problem. So with a higher-level controller setup (like outputting position deltas and letting a PID handle the rest) the RL agent would only have to learn the more interesting to look at part of this.
• By low-level problem I mean the "how do I spin these four motors so the drone doesn't flip over?" (stabilization) problem
• By credit assignment problem I mean a case such as this one: say the drone crashes, was it because the high-level decision was bad, meaning that the drone went the wrong direction or was it because the low-level control was bad, meaning that the motors weren't balanced?
• Even with this, it's been done before so I think the main thing to look at would be making the problem easier for the policy through better action parameterization maybe (or observations or reward funcion ) or again, trying to switch to the PID method? :)

Since the answer is that it is possible, I will also try to give an answer for 2 (but not for 3 since it's the case of infeasibility while I do think it is feasible) - Answer 2 will be posted as another comment because it will be quite more technical :)

[jediPhD]

Thanks for the detailed answer. I'm actually already using position delta as my plan B, so your suggestion aligns with what I'm currently doing. I'm also working on sensor fault detection and tracking control, so the credit assignment problem gets even more entangled with low‑level RPM control. Appreciate the help!

[visionofdavinci]

NOTE: for these proposals of what "key" tricks are I will cite papers whenever possible so that you can double check the information and read more on the implementation if needed :))

Question 2: If yes, what are the key tricks (action scaling, reward shaping, simulation parameters) that make it work?
Answer:

1. Action parameterization

• Instead of mapping [-1,1] to [0, max_rpm] or smaller ranges such as [-0.5, 0.5] try to parameterize the actions as deltas around hovers RPMs. there is a broader principle about action space design for learned quadrotor controllers I read about that you could tie this with. Kaufmann et al. (2022) [1] performed the first systematic benchmark of different action abstractions for learned quadrotor policies and found that policies commanding collective thrust + body-rates (CTBR) transferred to real hardware far more robustly than policies directly outputting individual rotor thrusts (SRT). The reason is that SRT requires the policy to implicitly learn the mapping from desired body torques to individual motor commands, which is sensitive to modelling errors.
  => So based on this I would assume that even if you stay with direct RPM control (SRT) in simulation, parameterizing around hover would reduce the space of actions the policy needs to explore, achieving a similar effect to what CTBR gives you structurally.
  [1] Kaufmann, E., Bauersfeld, L., & Scaramuzza, D. (2022). A Benchmark Comparison of Learned Control Policies for Agile Quadrotor Flight. IEEE ICRA 2022. (https://arxiv.org/abs/2202.10796)

• The code would look some thing like:
  hover_rpm = np.sqrt(self.M * self.G / (4 * self.KF))
action_rpm = hover_rpm + delta * rpm_range

  • with M as the mass, G as the acceleration due to gravity and KF as the motor constant or the propeller constant :)

2. Observations

• The main idea here is that you make sure that angular velocities are included in the observation space. If your observation only has roll/pitch/yaw but not angular rates (p, q, r), the policy has no way to anticipate rotational dynamics. It's essentially trying to do PD control with only the P term. The full state [x, y, z, roll, pitch, yaw, vx, vy, vz, p, q, r] should all be in there. Hwangbo et al. (2017) [2] demonstrated direct state-to-actuator mapping for a real Crazyflie-class quadrotor using RL, and their observation space includes the full state: position, orientation, and critically, angular velocities. Their cost function also directly penalizes angular velocity magnitude, on the idea that the policy can't learn to damp oscillations if it can't observe them. With this implementation they showed stable recovery even from being thrown upside-down which should be strong enough evidence that the full state observation is both necessary and sufficient for dorect low-level control.
  [2] Hwangbo, J., Sa, I., Siegwart, R., & Hutter, M. (2017). Control of a Quadrotor with Reinforcement Learning. IEEE Robotics and Automation Letters, 2(4), 2096–2103. (https://arxiv.org/abs/1707.05110)

3. Reward shaping

• I also got stuck here, but I can give you a paper that I found that helped me in understanding more the direction to go in. I am not sure what reward you are using, but from experience a reward that only penalizes tilt angle gives the policy no gradient signal about becoming tilted until it's already past the point of no return. I found that something along the lines of works fairly well:
  reward = (
- w1 * (roll**2 + pitch**2) # to penalize tilt
- w2 * (p**2 + q**2) # to penalize angular velocity
- w3 * np.var(rpm_actions) # to penalize asymmetric motor commands
+ w4 * alive_bonus # to reward not crashing per step
- w5 * altitude_error**2 # to task reward
)

• The angular velocity penalty is very important in early training specifically because without it, the only signal about instability comes from the tilt penalty, but by the time tilt is large enough to produce a meaningful penalty, the angular momentum is already too high to recover from. The np.var(rpm_actions) term should encourage symmetric motor outputs, which biases the policy toward stable configurations.

• Koch et al. (2018) [3] (the GymFC paper) studied reward design for RL-based quadrotor attitude control and showed that PPO was the only algorithm that consistently achieved stability across different reward formulations, but reward structure still mattered a lot a lot, in particular the inclusion of angular velocity penalties. They also found that including angular velocity in both the observation and reward was extremely important for convergence.
  [3] Koch, W., Mancuso, R., West, R., & Bestavros, A. (2018). Reinforcement Learning for UAV Attitude Control. ACM Transactions on Cyber-Physical Systems, 3(2). (https://arxiv.org/abs/1804.04154)

I know this is a super lengthy answer but I wanted to make sure I gave proper references and information! Hope it helps and would be happy to discuss further about this if needed! :)

[JacopoPan]

Thanks for the lively conversation @visionofdavinci @jediPhD

I largely agree with @visionofdavinci 's point (this was indeed one of the things that this repo wanted to highlight)

To answer shortly yes, but it is way harder than using a high-level controller. The difficulty of this question is coming from the fact that you are asking the policy to learn both low-level stabilization and task-level behaviour since you are using direct RPM control, which creates a much harder credit assignment problem.

I should clarify w.r.t. this point

Instead of mapping [-1,1] to [0, max_rpm] or smaller ranges such as [-0.5, 0.5] try to parameterize the actions as deltas around hovers RPMs.

that this is already part of the action pre-processing in BaseRLAviary.py

https://github.com/utiasDSL/gym-pybullet-drones/blob/82b0fda6d005b287cdc1ef0c313d09a361027c45/gym_pybullet_drones/envs/BaseRLAviary.py#L191-L192