[serge] integration failure triage - 2026-10-29 #50143

Automated integration-failure triage for the daily CI window 2026-10-09 → 2026-10-29.

This issue was generated by AI-assisted automation. The grouping, summaries, and recommended follow-up can be incomplete or misleading; verify the failures before acting.

Serge dispatched one task per failure group below — each opens or updates its own PR on a serge/fix/itf-<fingerprint> branch. This table is refreshed in place as Serge runs: a group links its #<pr> when opened, shows 🚫 no fix when Serge found no safe change, ⚠️ task failed on error, or (pending) while still running (a late PR links on the next nightly run).

The Task column links Serge's own page for the run — the steps it went through, what each tool cost, and the full task text. 🔒 means it is not public: the host is VPN-internal and the page needs a Serge login. A task links once Serge has accepted it, so the column fills in on the first refresh after dispatch.

Dispatched failure groups

Not dispatched — environment / dependency

These groups were triaged but NOT handed to Serge: their failure mode is a property of the runner or the environment, so no minimal source patch can fix them. They need a human (runner capacity, a dependency pin).

2 models ran out of device memory (5 failures) — needs runner capacity, not a patch, so none of these were dispatched: llama4 (3), exaone4_5 (2).

5 group(s) skipped — stopped failing before this run (generation, minimax_m3_vl, minicpmv4_7, moshi +1 more). Not dispatched and not awaiting a human.

Outcome recap

Why each group that opened no PR ended without one, and what it cost — surfaced here from the Serge dashboard. Failing tests links straight to each test's dashboard view, since these are the groups a human has to pick up.

Unverified branches — how to finish them

Serge committed a patch for each group below but the GPU gate never judged it, so no PR was opened and the branch was kept. Check the already-dispatched run before re-running: it often completes after Serge stops waiting, and reading its artifact is free where a re-run is not.

mistral4 — unverified patch on serge/fix/itf-cb09ea6673d1-bf12d778 (4dcc377f74cc)

# 1. The verify run Serge dispatched may have finished after it gave up —
#    check that run (URL in the Reason cell above) before spending GPU time:
#    gh run download <run-id> -R huggingface/transformers -n serge-verify-result-aws-g5-12xlarge-cache

# 2. Only if no verdict exists, re-run it:
gh workflow run serge-verify-caller.yml -R huggingface/transformers \
  -f mode=verify \
  -f base_sha=$(gh api repos/huggingface/transformers/commits/4dcc377f74cccc618de981e3585da114bfcb17f0 --jq '.parents[0].sha') \
  -f commit_sha=4dcc377f74cccc618de981e3585da114bfcb17f0 \
  -f test_nodeids='tests/models/mistral4/test_modeling_mistral4.py::Mistral4IntegrationTest::test_mistral_small_4_generation tests/models/mistral4/test_modeling_mistral4.py::Mistral4IntegrationTest::test_mistral_small_4_logits' \
  -f model=mistral4 \
  -f machine_type=aws-g5-12xlarge-cache

cohere2_vision — unverified patch on serge/fix/itf-87028e319b86-fa6c1a77 (caa7967b1948)

# 1. The verify run Serge dispatched may have finished after it gave up —
#    check that run (URL in the Reason cell above) before spending GPU time:
#    gh run download <run-id> -R huggingface/transformers -n serge-verify-result-aws-g5-12xlarge-cache

# 2. Only if no verdict exists, re-run it:
gh workflow run serge-verify-caller.yml -R huggingface/transformers \
  -f mode=verify \
  -f base_sha=$(gh api repos/huggingface/transformers/commits/caa7967b1948640716b42f222c77056f85ea7b29 --jq '.parents[0].sha') \
  -f commit_sha=caa7967b1948640716b42f222c77056f85ea7b29 \
  -f test_nodeids='tests/models/cohere2_vision/test_modeling_cohere2_vision.py::Cohere2IntegrationTest::test_model_integration_batched_generate tests/models/cohere2_vision/test_modeling_cohere2_vision.py::Cohere2IntegrationTest::test_model_integration_forward' \
  -f model=cohere2_vision \
  -f machine_type=aws-g5-12xlarge-cache

Generated 2026-10-30T00:54:39+00:00.