Repository navigation
[serge] integration failure triage - 2026-11-01 #50231
Description
Automated integration-failure triage for the daily CI window 2026-10-12 → 2026-11-01.
This issue was generated by AI-assisted automation. The grouping, summaries, and recommended follow-up can be incomplete or misleading; verify the failures before acting.
Serge dispatched one task per failure group below — each opens or updates its own PR on a serge/fix/itf-<fingerprint> branch. This table is refreshed in place as Serge runs: a group links its #<pr> when opened, shows 🚫 no fix when Serge found no safe change, ⚠️ task failed on error, or (pending) while still running (a late PR links on the next nightly run).
The Task column links Serge's own page for the run — the steps it went through, what each tool cost, and the full task text. 🔒 means it is not public: the host is VPN-internal and the page needs a Serge login. A task links once Serge has accepted it, so the column fills in on the first refresh after dispatch.
Dispatched failure groups
| Model | Error | Occurrences | PR | Task |
|---|---|---|---|---|
pocket_tts (regressed by PR #49589) |
mixed — other (14), list output differs (2) | 16 | 🚫 no fix | 🔒 task |
gemma2 (regressed by PR #49811) |
mixed — list output differs (2) | 2 | #50098 | 🔒 task |
cohere2_moe |
other — other (8) | 8 | 🚫 no fix | 🔒 task |
bridgetower |
other — other (6) | 6 | 🚫 no fix | 🔒 task |
blt |
import_or_config — other (4) | 4 | 🚫 no fix | 🔒 task |
kimi_k25 |
cuda_runtime — other (2) | 2 | 🚫 no fix | 🔒 task |
ovis2 |
output_mismatch — list output differs (2) | 2 | #50233 | 🔒 task |
reformer |
output_mismatch — other (2) | 2 | #49841 | 🔒 task |
exaone4 |
OOM — other (1) | 1 | 🚫 no fix | 🔒 task |
qwen3_vl_moe |
OOM — other (1) | 1 | #50148 | 🔒 task |
Not dispatched — environment / dependency
These groups were triaged but NOT handed to Serge: their failure mode is a property of the runner or the environment, so no minimal source patch can fix them. They need a human (runner capacity, a dependency pin).
1 model ran out of device memory (3 failures) — needs runner capacity, not a patch, so none of these were dispatched: llama4 (3).
5 group(s) skipped — stopped failing before this run (generation, minimax_m3_vl, kosmos2, moshi +1 more). Not dispatched and not awaiting a human.
Outcome recap
Why each group that opened no PR ended without one, and what it cost — surfaced here from the Serge dashboard. Failing tests links straight to each test's dashboard view, since these are the groups a human has to pick up.
| Group | Reason | LLM | Tokens (in / out) | Failing tests |
|---|---|---|---|---|
pocket_tts (regressed by PR #49589) |
All 16 failures trace to artifacts outside this repo, not to library code. | deepseek-ai/DeepSeek-V4.1-Flash-1017:together |
1,739,289 / 38,601 | test_drifting · test_training · test_voice_cloning · +5 more |
cohere2_moe |
All 8 Cohere2MoeIntegrationTest failures crash in from_pretrained with SafetensorError: Error while deserializing: header too small (reported at line 386, which is inside the loader, not the 197-line test file). The test never reaches its assertions, so no expected value is wrong. |
deepseek-ai/DeepSeek-V4.1-Flash-1017:together |
2,094,512 / 95,800 | test_generation_beyond_sliding_window_1_sdpa · test_generation_beyond_sliding_window_2_eager · test_model_bf16 · +1 more |
bridgetower |
[not_fixed] GPU verification did not confirm the fix (not_fixed): the patch did not turn the targeted tests green. |
deepseek-ai/DeepSeek-V4.1-Flash-1017:together |
1,537,300 / 91,492 | test_constrastive_learning · test_image_and_text_retrieval · test_masked_language_modeling |
blt |
Root cause confirmed: BltModel.forward builds DynamicCache(config=self.config), and DynamicCache.__init__ calls get_text_config(decoder=True), which for BLT returns the top-level BltConfig — a config with no num_hidden_layers (only its nested encoder/decoder/global/patcher sub-configs have it). _PerLayerConfigView.__len__ then raises the AttributeError. |
deepseek-ai/DeepSeek-V4.1-Flash-1017:together |
887,023 / 15,906 | test_model_logits · test_model_logits_bf16 |
kimi_k25 |
[not_fixed] GPU verification did not confirm the fix (not_fixed): the patch did not turn the targeted tests green. |
deepseek-ai/DeepSeek-V4.1-Flash-1017:together |
1,128,913 / 150,045 | test_model_logits · test_model_logits_batched |
exaone4 |
No fix was proposed. | deepseek-ai/DeepSeek-V4.1-Flash-1017:together |
262,478 / 57,049 | test_model_generation_beyond_sliding_window |
Generated 2026-11-02T00:29:58+00:00.
Activity
github-actions commentedon Nov 2, 2026
Superseded by #50280 — closing this stale nightly integration-failure triage issue. Open PRs stay linked from the current issue.