[3outeille]

What does this PR do?

Stack: #50493 → #50496 → #50511 → this PR → #50515 → #50499 → #50521 → #50522

Context parallelism now composes with expert parallelism: DistributedConfig(cp_size=C, ep_size=E, ...). Only pp_size is still refused with cp_size.

• Token dispatch (ep_dispatch_experts, the default of most MoE models): each rank dispatches the tokens of its own cp slice. A cp slice plays the role a batch plays for the expert mesh. efsdp = fsdp * cp * tp / ep already covers the world.
• Expert gradients: set_gradient_divide_factor(dp_size) already gives the gradient of the full-sequence loss, because dp_size includes cp_size and the CP loss hook scales each rank's gradient by cp_size. No code change; only the comment is updated.
• The all-reduce expert plans still need ep_size=tp_size, and the ranks of a tp group hold the same slice.

Tests

tests/context_parallel, CPU (gloo), tiny Qwen3-MoE. Loss, logits slice and every gradient, experts included, match an unsharded run:

setup	plan	ranks
cp=2 × ep=2	token dispatch	2
cp=2 × fsdp=2 × ep=4	token dispatch	4
cp=2 × fsdp=2 × ep=2 (efsdp=2)	token dispatch	4
cp=2 × tp=2 × ep=4	token dispatch	4
cp=2 × dp_replicate=2 × ep=2	token dispatch	4
tp=2 × cp=2 × ep=2	all-reduce	4

The existing EP / FSDP tests of qwen3_moe and tests/expert_parallel pass.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.