SACD Discrete Soft Actor Critic

[splatter96]

This PR introduces the Soft Actor Critic for discrete actions (SACD) algorithm.

Description

This PR implements the SAC-Discrete algorithm as described in this paper https://arxiv.org/abs/1910.07207. This implementation borrows code from the papers original implementation (https://github.com/p-christ/Deep-Reinforcement-Learning-Algorithms-with-PyTorch) as well as provided by the issues author who requested this feature in stable baselines (https://github.com/toshikwa/sac-discrete.pytorch)

Context

• I have raised an issue to propose this change (required)
• Original issue in the stable baselines repo [Feature request] Implement SAC-Discrete DLR-RM/stable-baselines3#157

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)

Checklist:

• I've read the CONTRIBUTION guide (required)
• The functionality/performance matches that of the source (required for new training algorithms or training-related features).
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have included an example of using the feature (required for new features).
• I have included baseline results (required for new training algorithms or training-related features).
• I have updated the documentation accordingly.
• I have updated the changelog accordingly (required).
• I have reformatted the code using make format (required)
• I have checked the codestyle using make check-codestyle and make lint (required)
• I have ensured make pytest and make type both pass. (required)

Note: we are using a maximum length of 127 characters per line

[araffin]

Hello,
thanks for the PR =)

The functionality/performance matches that of the source (required for new training algorithms or training-related features).

please don't forget that part (see contributing guide).
I think there are discussion about the results here too: vwxyzjn/cleanrl#270

[splatter96]

Hello,
thanks for the feedback :)
Sorry for the late reply! Should I add the performance comparison to the source similarly as it is done in the official stable baselines3 algorithm pages? As in create a baselines3-zoo config for it and add the plots to this PR?

[araffin]

yes please =)

[daniel-redder]

Hello, thanks for the feedback :) Sorry for the late reply! Should I add the performance comparison to the source similarly as it is done in the official stable baselines3 algorithm pages? As in create a baselines3-zoo config for it and add the plots to this PR?

Do you have the performance results for this? I came across this PR looking for implementations on SACD. Thank you.

[splatter96]

Hello, thanks for the feedback :) Sorry for the late reply! Should I add the performance comparison to the source similarly as it is done in the official stable baselines3 algorithm pages? As in create a baselines3-zoo config for it and add the plots to this PR?

Do you have the performance results for this? I came across this PR looking for implementations on SACD. Thank you.

Unfortunately I never found the time to do the performance benchmark. I however use this implementation in several projects of mine with good results. So the implementation seems to be correct.

[gyunt]

It seems self.features_extractor is None is not handled in forward method.

[alektebel]

@araffin @splatter96

Hi, I’d like to contribute to this PR by adding the rl-baselines3-zoo benchmarks.

Plan:

• Run 5 seeds each on:
  • CartPole-v1 (MlpPolicy)
  • MsPacmanNoFrameskip-v4 (CnnPolicy)
  • BreakoutNoFrameskip-v4 (CnnPolicy)
• Train for 1 million timesteps per environment
• Generate the standard zoo learning-curve plots (mean ± std)
• Compare visually/qualitatively against the original p-christ and toshikwa results

I already tested the PR locally on CartPole and it trains perfectly (reaches ~250 reward very fast).

Before I run the full 5-seed Atari experiments, could you confirm the environments and timestep budget look good, or would you prefer anything different?

I can have the results ready within the next few days and either post the plots here or open a small follow-up PR in rl-baselines3-zoo if that’s cleaner.

Thanks!

[araffin]

Hello,

sorry for the late reply.

Plan:

That would be a good start. Please use the RL Zoo for that (and the wandb integration if possible).

I already tested the PR locally on CartPole and it trains perfectly (reaches ~250 reward very fast).

250 is not very good for CartPole (it should reach ~500).

[alektebel]

Hi @araffin ,

On a second run, CartPole reached 500 reward after about 100k steps, so it seems promising. I'll follow the plan then and use rl zoo and wandb integration for the experiment result logs, then compare the learning curves between the algorithms and contrast it against the original paper. In the potential case the performance of this PR does not reach the expected value, I'll debug it myself in order to understand why.