GitHub · Where software is built

Labels Milestones New issue

Minimum number of GPUs needed to pretrain llama4_17bx16e - 8 ?

#1543

· githubsgi opened

on Aug 6, 2025

Transformer is running with float32 instead of bfloat16 !

#1525

· githubsgi opened

on Aug 4, 2025

Weight tying between embedding and LM head layer

#1524

· Luosuu opened

on Aug 4, 2025

MOE - grouped gemm is not supported on AMD ROCM

#1523

· AlirezaShamsoshoara opened

on Aug 4, 2025

[DSV3][CP][AC] 'nan' in grad after attention op in backward when CP and full AC are used at the same time

module: context parallel

#1522

· XilunWu opened

on Aug 4, 2025

PP Stage init hangs on multi-nodes

#1492

· rakkit opened

on Jul 30, 2025

Is there documentation on what exactly are 'dp_shard_mod_ep' and 'dp_shard_in_ep'] ?

#1482

· githubsgi opened

on Jul 29, 2025

Is FSDP+TP+EP supported for Llama4 ?

#1478

· githubsgi opened

on Jul 28, 2025

[Question] TP only + amp by <code>fully_shard(..., ignored_params=...)</code>

#1469

· junjzhang opened

on Jul 28, 2025

possible memory leaking of DP2EP with recompute

#1467

· rakkit opened

on Jul 26, 2025

CP + SDPA issue w/ SAC when using TorchFunctionMode

module: context parallel

#1436

· XilunWu opened

on Jul 21, 2025

CP + SDPA fails with type conversion error at profiling step when <code>deterministic=True</code>

module: context parallel

#1435

· XilunWu opened

on Jul 21, 2025