Is FSDP+TP+EP supported for Llama4 ?

[githubsgi]

Wondering if FSDP+TP+EP is supported for pre-training LLama4 ?

[tianyu-l]

It depends on your definition of EP.
For "dp2ep" EP, it's supported (#1324), although there seems to be a memory leakage issue with EP + activation checkpoint.
There's a temporary fix from #1467

We are working on debugging this issue.