[ArthurZucker]

One right-padded forward per request; the token before each mark is read with slice_logits_to_keep, for decision models and generic chat models.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

Speedup on vllm-sr/d3 (27B Qwen3.5 decider), A100 80GB, bf16, median of 10. One prompt per question: one forward per prompt (before) vs one padded batch (this PR).

questions	tokens	sdpa before	sdpa after	sdpa speedup	FA2 before	FA2 after	FA2 speedup
4	481	504 ms	160 ms	×3.16	535 ms	165 ms	×3.25
16	1924	2019 ms	566 ms	×3.57	2122 ms	569 ms	×3.73
64	7696	8340 ms	2174 ms	×3.84	8449 ms	2189 ms	×3.86

FA2 is kernels-community/flash-attn2. Batched vs per-prompt scores differ by ≤0.125 (sdpa) / ≤0.0625 (FA2) in bf16 logits. causal_conv1d was not installed, so both columns use the torch fallback for the linear-attention conv.

[ydshieh]

test_deciders still runs out of memory on the 24 GB A10G CI runners: the deciders are 27B checkpoints (~54 GB in bf16). Loading them with device_map="auto" fixes it. On an A10G, all three sub-tests pass in 271 s. The change, from #50588:

# tests/test_decision_head_common.py, in check_decision
placement = {} if "device_map" in pipeline_kwargs else {"device": torch_device}
decide = pipeline(..., **placement, **pipeline_kwargs)  # instead of device=torch_device

# tests/models/jev/test_modeling_jev.py
def test_deciders(self):
    for repo, expected in DECIDERS.items():
        with self.subTest(repo):
            check_decision(self, None, repo, expected, atol=1e-2, device_map="auto")
        cleanup(torch_device, gc_collect=True)

[ydshieh]

@ArthurZucker test_jev_35b fails on the daily CI with an OOM, but fixing the OOM (e.g. device_map="auto") isn't enough: hf-internal-testing/JEV-35B contains only the head (model.safetensors has just proj.weight and proj.bias), so every backbone weight is missing from the checkpoint and gets randomly initialized. Could you take a look at how to adjust the test?

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: jev

[LysandreJik]

Love it!

[LysandreJik]

This only looks at the first input for its keys, but it should arguably look at all of them?

Right now if a single input has an image and it is the first of the batch, then it will fail on the following ones. If it's not the first of the batch, then that input's image will be skipped.

[LysandreJik]

fun gotcha here: if a processor's output's last dimension can equal the length of the first prompt, then this won't work well as it won't concatenate stuff together but instead treat it as per-token and pad it, keeping only its first row per prompt.

example of where it fails:

import torch
from transformers.decision.modeling_utils import batch_prompts

def prompt(seq_len):
    return {"input_ids": torch.ones(1, seq_len, dtype=torch.long), "marks": torch.tensor([seq_len]),
            "pixel_values": torch.randn(4, 1176)}

print(batch_prompts([(prompt(1175), []), (prompt(1175), [])])["pixel_values"].shape)  # (8, 1176): correct
print(batch_prompts([(prompt(1176), []), (prompt(1176), [])])["pixel_values"].shape)  # (2, 1176): incorrect

(1176 was chosen here as that's the value Qwen2.5 VL uses)

[github-actions]

This PR might need docs.

DecisionMixin.decide now takes logits_to_keep instead of marks, expects decision_questions as a list of lists (one per prompt) and returns a list of lists of scores, so the documented example model.decide(**inputs, decision_questions=questions) in the JEV page no longer works and must be rewritten to use batch_prompts and the batched signature. The pipeline(..., trust_remote_code=False) change (using the built-in pipeline instead of a repo's custom one for supported tasks) is also worth a line.

Pages that might need an update:

• docs/source/en/model_doc/jev.md
• docs/source/en/main_classes/pipelines.md

_{Posted automatically after approval. Docs live in docs/source/en/, and updating them turns the docs-check check green. The check doesn't block merging.}