[sergereview]

Original CI failure

• Failure group: 2 integration tests for model ovis2failing withoutput_mismatch (list output differs (2))
• tests/models/ovis2/test_modeling_ovis2.py::Ovis2IntegrationTest::test_small_model_integration_test_batch_different_resolutions [multi-gpu] (output_mismatch, seen 21/21)

CI traceback — tests/models/ovis2/test_modeling_ovis2.py::Ovis2IntegrationTest::test_small_model_integration_test_batch_different_resolutions

(line 312)  AssertionError: Lists differ: ['sys[81 chars]ant\n', 'system\nYou are a helpful assistant.\[139 chars]et.'] != ['sys[81 chars]ant\nAnswer: I see a brown dog standing on a w[224 chars]et.']

Where to watch it:

• test_small_model_integration_test_batch_different_resolutions — Test dashboard

Relates to #50231

test_small_model_integration_test_batch_different_resolutions batches a 1-patch image (320x240, 256 image tokens) with a 5-patch one, so the shorter sample is padded. With the tokenizer's default right padding, generation for that sample starts from a pad token and emits EOS immediately, which is the empty assistant\n output in the trace.
The test now sets padding_side = "left" in setUp, as sibling VL integration tests do (e.g. tests/models/hunyuan_vl/test_modeling_hunyuan_vl.py:305), so both samples generate from their real last token.
The low-res expectation is updated to the left-padded output: same description of the same dog image, without the Answer: prefix and two tokens further into the sentence within max_new_tokens=20. This is a padding-side change, not hardware drift, so no device key is added.

---

✅ Verified on GPU

serge ran the targeted @slow test(s) on a GPU runner and opened this PR only after they passed with this patch. Each targeted test was run 5× on both the pre-patch and patched trees to rule out flakiness — the result held on every run.

• Passes with this patch: https://github.com/huggingface/transformers/actions/runs/41367070817
• Failed before the fix (base commit): https://github.com/huggingface/transformers/actions/runs/41361144835

Possibly related

Existing issues/PRs mentioning test_small_model_integration_test_batch_different_resolutions (keyword match — not verified to share a root cause):

• #41095 — Add LLaVA-OneVision-1.5 model and related configurations (PR, open, updated 2026-10-25)
• #49301 — [Qwen3VLMoe] Update expected text in test_small_model_integration_test_expand (PR, closed, updated 2026-10-08)
• #49116 — qwen3 models map to wrong tokenizer class on the hub (PR, closed, updated 2026-09-28)
• #49011 — Added more usage of MemoryCleanupMixin (PR, closed, updated 2026-09-22)
• #48589 — Synthetic test assets (PR, closed, updated 2026-09-10)

---

This change was produced automatically by serge from a CI failure report. The patch was generated by an LLM and applied by serge; review before merging.

serge v0.1.0 · model: deepseek-ai/DeepSeek-V4.1-Flash-1017:together · 35 LLM turns · 54 tool calls · 243.3s · 1624850 in / 80284 out tokens

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: ovis2

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[zucchini-nlp]

my two cents: it is very weird that we keep seeing random failures due to padding side, not the first time 🤔

Could it be smth on tokenizer classes or are we sure that test never passed? Maybe we need some way to catch these random default value changes in test_tokenization or smth if it happens so often