[3outeille]

What does this PR do?

Stack: #50493 → #50496 → #50511 → #50514 → this PR → #50499 → #50521 → #50522

With context parallelism, a sequence length that cp_size does not divide used to raise. Now ModelContextParallel, the style of the model itself (#50493), pads the full sequences at the end, before the model builds its masks:

• input_ids get pad_token_id (or 0), and inputs_embeds get zeros. No real token attends to the padding, because it comes after them.
• position_ids, if given, continue those of the last document.
• A 2D attention_mask, if given, gets zeros.
• labels and shift_labels get -100, so num_items_in_batch still counts the real tokens only.
• outputs.logits is this rank's slice of the padded sequence.

Tests

tests/context_parallel, CPU (gloo):

• length 7 with cp=2 (plain, packed, inputs_embeds, with a padding mask) and length 5 with cp=4;
• the loss, every gradient, and the logits at the real positions match an unsharded run on the unpadded sequences;
• a single-process test of what ModelContextParallel pads.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.