Skip to content

Releases: NVIDIA/Megatron-LM

NVIDIA Megatron Core 0.14.0rc4

04 Aug 04:12
@ko3n1g ko3n1g
core_v0.14.0rc4
7f7439f
Compare
Choose a tag to compare
NVIDIA Megatron Core 0.14.0rc4 Pre-release
Pre-release

Prerelease: NVIDIA Megatron Core 0.14.0rc4 (2025-08-04)

Loading

NVIDIA Megatron Core 0.14.0rc3

28 Jul 04:13
@ko3n1g ko3n1g
core_v0.14.0rc3
adbab64
Compare
Choose a tag to compare
NVIDIA Megatron Core 0.14.0rc3 Pre-release
Pre-release

Prerelease: NVIDIA Megatron Core 0.14.0rc3 (2025-07-28)

Loading

NVIDIA Megatron Core 0.13.0

25 Jul 18:04
@ko3n1g ko3n1g
core_v0.13.0
c550cf6
Compare
Choose a tag to compare
NVIDIA Megatron Core 0.13.0 Latest
Latest
  • Support bf16 dtype for optimizer states to use precision-aware optimizer in TransformerEngine
  • MoE
    • Features:
      • Flexible Asymmetric Virtual Pipeline Parallelism with Custom Pipeline Layout (--pipeline-model-parallel-layout)
      • Add support to pass custom parallelism groups to MoE modules.
      • Add Hybrid Shard Data-Parallel support for MoE models (--num-distributed-optimizer-instances)
      • Support EP + custom FSDP training for DeepSeek-V3
      • FP8 support for Multi-Token-Prediction
    • Memory Optimization
      • Fine-grained recomputation to reduce activation memory. (--recompute-modules with --recompute-granularity selective)
      • Memory efficient token permutation by moving the probs multiplication from unpermutation to activation function of GroupedMLP.
    • Performance Optimization
      • MLA RoPE fusion kernel and YARN embedding cache.
      • FP8 padding optimization of MoE models by padding the routing map.
    • Bug fixes:
      • Fix the aux loss calculation when expert_bias or group limited routing is used. This leads to load_balancing_loss values change compared to the previous version.
      • Fix packed sequence support for MLA
    • Known Issues:
      • MTP is not compatible with flexible pipeline layout, will be fixed at !3594.
      • MTP convergence issue with TP2, will be fixed at !3594.
Assets 2
Loading

NVIDIA Megatron Core 0.14.0rc2

21 Jul 04:12
@ko3n1g ko3n1g
core_v0.14.0rc2
db41707
Compare
Choose a tag to compare
NVIDIA Megatron Core 0.14.0rc2 Pre-release
Pre-release

Prerelease: NVIDIA Megatron Core 0.14.0rc2 (2025-07-21)

Loading

NVIDIA Megatron Core 0.13.0rc4

22 Jul 08:03
@ko3n1g ko3n1g
core_v0.13.0rc4
ac198fc
Compare
Choose a tag to compare
NVIDIA Megatron Core 0.13.0rc4 Pre-release
Pre-release

Prerelease: NVIDIA Megatron Core 0.13.0rc4 (2025-07-22)

Loading

NVIDIA Megatron Core 0.13.0rc3

17 Jul 15:04
@ko3n1g ko3n1g
core_v0.13.0rc3
9b9ea83
This commit was signed with the committer’s verified signature.
ko3n1g oliver könig
GPG key ID: 2A0D811D627CDD85
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
NVIDIA Megatron Core 0.13.0rc3 Pre-release
Pre-release

Prerelease: NVIDIA Megatron Core 0.13.0rc3 (2025-07-17)

Loading

NVIDIA Megatron Core 0.14.0rc1

14 Jul 04:12
@ko3n1g ko3n1g
core_v0.14.0rc1
44fa0ea
Compare
Choose a tag to compare
NVIDIA Megatron Core 0.14.0rc1 Pre-release
Pre-release

Prerelease: NVIDIA Megatron Core 0.14.0rc1 (2025-07-14)

Loading

NVIDIA Megatron Core 0.14.0rc0

07 Jul 04:10
@ko3n1g ko3n1g
core_v0.14.0rc0
8f4d909
Compare
Choose a tag to compare
NVIDIA Megatron Core 0.14.0rc0 Pre-release
Pre-release

Prerelease: NVIDIA Megatron Core 0.14.0rc0 (2025-07-07)

Loading

NVIDIA Megatron Core 0.13.0rc2

02 Jul 21:36
@ko3n1g ko3n1g
core_v0.13.0rc2
001d15c
Compare
Choose a tag to compare
NVIDIA Megatron Core 0.13.0rc2 Pre-release
Pre-release

Prerelease: NVIDIA Megatron Core 0.13.0rc2 (2025-07-02)

Loading

NVIDIA Megatron Core 0.13.0rc1

03 Jul 07:16
@ko3n1g ko3n1g
core_v0.13.0rc1
7da88d7
Compare
Choose a tag to compare
NVIDIA Megatron Core 0.13.0rc1 Pre-release
Pre-release

Prerelease: NVIDIA Megatron Core 0.13.0rc1 (2025-07-03)

Loading