Notable performance regression between version 0.2.0(?) and latest master branch code

[QuantumToasted]

Hello,

When I shelved my massive, broken cleanup PR, I set out to try again, with smaller steps, more commits, more testing, to make sure I didn't break anything.

My very first move was to simply remove the NuGet dependencies within the library - to strictly make them reference each other as projects and allow more tight coupling and rapid development. After doing so, I observed an extremely sharp decline in the ability for the CartPole-v1 example to "learn" and make meaningful progress, even with the exact same hyperparameters. I am unsure exactly which version of RLMatrix this regression became noticeable. My memory is a little foggy because I performed this test several weeks ago, but I believe as the current master branch code references a 0.4.x version, I simply randomly chose an older version, 0.2.0, and this issue went away.

I wrote a very simple example demonstrating this issue on the nouveau-2.0 of my fork of RLMatrix. A video demonstrating the situation can be found below. Simply changing Old to false in CartPole-v1.csproj creates a stark contrast in performance. I would be happy to provide relevant logs or other debugging information if needed, finding details that could explain this is a little out of my range of expertise.

2025-03-18.19-54-57-00.00.00.000-00.01.40.967.mp4

There absolutely could be something I'm missing here! I'm hoping that maybe I'm misusing the newer code in some way that makes it inconsistent with the older code.

[asieradzk]

Hey. I wouldn't say this is a learning regression. its just that deep reinforcement learning can be incredibly fragile.

I remember somewhere between 0.2 and 0.4 changing how we interpret the batch size and learning horizon for the PPO. I am almost sure in the version 0.2 we count the number of steps and in 0.4 its episodes.

The reason being is that PPO is on-policy algorithm and training will become unstable if you have experiences enter the buffer that were generated by the previous version of policy. However if the batchSize was counted based on "steps" you will frequently run into situation where policy is swapped mid-episode generating less meaningful experience.

With that said try a lot smaller batch size:

var optsPPO = new PPOAgentOptions(
   batchSize: 4,        // Number of experiences used per training step
   memorySize: 10000,     // Total experiences stored for training
   gamma: 0.99f,          // Reduces importance of future rewards
   gaeLambda: 0.95f,      // Balances bias-variance in advantage estimation
   lr: 1e-3f,             // Step size for gradient descent
   width: 128,            // Neurons per hidden layer
   depth: 2,              // Number of hidden layers
   clipEpsilon: 0.2f,     // Limits policy update size
   vClipRange: 0.2f,      // Limits value function update
   cValue: 0.5f,          // Adjusts importance of value function loss
   ppoEpochs: 3,          // Training iterations per batch
   clipGradNorm: 0.5f,    // Prevents extreme gradient updates
   entropyCoefficient: 0.005f,  // Encourages exploration
   useRNN: false           // Enables memory for temporal dependencies
);

This works for me on the newest RLMatrix with ppo. can possibly go to 16 batch size or even higher if you tweak other hyperparameters.

tl;dr;
You cant change from old PPO to new PPO without reducing the batch size.

...Also keep in mind that 0,4 version is quite a lot faster in terms of FPS so make sure to check the actual numerical values rather than realtime episode length when comparing between them.

[asieradzk]

I think we should maybe have two PPO implementations. The one that gave you issues is a bit modified compared to literature PPO. I made it that way to make it more friendly for distributed training but because of it the trivial application suffers...