Training/simulation crashing, possibly due to NaN values

[jackzeng-robotics]

Hi developers,

I am working on a setup where multiple drones carry an object together with strings. The strings are modeled as 7 thin links with ball joints between them. The ball joints are modeled as 3 continuous joints in x, y and z direction. It looks like this:

The links have a very low mass and inertia and before I thought this could create some stability issues. What I observed: given big forces and torques, the setup makes very wild movements and then the setup disappears from the simulation. Probably due to some NaN values. This also caused my training to crash. My hypothesis before was that the cable links have very low mass and inertia causing them to fly away when under high loads, for example when compressed. So, I added numerous termination terms that stopped this from happening, for example limiting the angle between the drone and the links, setting terminations on high velocity values, angular rates etc.

This pretty much solved the issue for a long time until I implemented a low level controller for the drones to go with it. The behaviour of the low level controller seems fine, similar to what we get in real life. Without the RL policy is does not exhibit any strange behaviour, moreover the actions (forces on the drone rotors) are clamped as well. Now, the error has come back and I get this error when training with SKRL:

[Error] [omni.physx.plugin] PhysX error: The application needs to increase PxGpuDynamicsMemoryConfig::foundLostAggregatePairsCapacity to -1423966208, otherwise, the simulation will miss interactions, FILE /builds/omniverse/physics/physx/source/gpubroadphase/src/PxgAABBManager.cpp, LINE 1269

And this when using RSL-RL:
File "/home/isaac-sim/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/distributions/normal.py", line 71, in sample
return torch.normal(self.loc.expand(shape), self.scale.expand(shape))
RuntimeError: normal expects all elements of std >= 0.0

Is there any way to figure out where the crash comes from? Like finding the last state of the drones in some log somewhere? Or do you have any idea on what it could be?

Thanks in advance!

Jack

[StrainFlow]

Have you reported this in the Issues section? That would be the better place to post a crash bug like this.

In order for them to solve this they'll need to be able to reproduce the issue, so the more info you have ready to provide the better.

[DavidTer756]

Hello Jack,

I hope you're doing well. I came across your work on a similar system in Isaac Lab and noticed that you had encountered an issue quite like the one I’m currently facing.

In my setup, I’ve implemented four drones carrying a cargo, with cables modeled as 13 linked segments connected by joints. During the simulation, the joints sometimes reach extremely high velocities, which causes instability and eventually results in NaN values.

I was wondering if you might be willing to share how you approached or resolved this issue. I would really appreciate any insights or guidance you could offer.

Thank you in advance for your time and help!

[jackzeng-robotics]

Hi David,

I do not have a clean answer for the problem unfortunately. It is however not an uncommon problem, especially in contact based rigid body simulations. I got pretty robust performance when I:

• Decreased the simulation timestep enough (I ran it at 200-300 Hz)
• Terminated the episode before aggressive situations could arise (terminate based on cable angle, payload angle, height, drone angle etc.. before the bodies start to collide with eachother)
• When the setup for my RL policies became better it also happened less since these dangerous were less common

Hope this helps!
Kind regards,
Jack

[DavidTer756]

Hi Jack,

I implemented some episode terminations before aggressive situations, and it helped a lot. Thank you very much for your help, I really appreciate it.

Best regards,
David

[jackzeng-robotics]

Glad to hear it helped!