Repository navigation
[serge] integration failure triage - 2026-11-13 #50742
Description
Automated integration-failure triage for the daily CI window 2026-10-24 → 2026-11-13.
This issue was generated by AI-assisted automation. The grouping, summaries, and recommended follow-up can be incomplete or misleading; verify the failures before acting.
Serge dispatched one task per failure group below — each opens or updates its own PR on a serge/fix/itf-<fingerprint> branch. This table is refreshed in place as Serge runs: a group links its #<pr> when opened, shows 🚫 no fix when Serge found no safe change, ⚠️ task failed on error, or (pending) while still running (a late PR links on the next nightly run).
The Task column links Serge's own page for the run — the steps it went through, what each tool cost, and the full task text. 🔒 means it is not public: the host is VPN-internal and the page needs a Serge login. A task links once Serge has accepted it, so the column fills in on the first refresh after dispatch.
Dispatched failure groups
| Model | Error | Occurrences | PR | Task |
|---|---|---|---|---|
pocket_tts (regressed by PR #49589) |
mixed — other (14), list output differs (2) | 16 | 🚫 no fix | 🔒 task |
vipllava (regressed by PR #49594) |
mixed — other (2) | 2 | 🔒 task | |
diffusion_gemma |
output_mismatch — tensor values differ (6), list output differs (2) | 8 | 🚫 no fix | 🔒 task |
starcoder2 |
output_mismatch — list output differs (6) | 6 | #47983 | 🔒 task |
blt |
import_or_config — other (4) | 4 | 🔒 task | |
qwen3_vl_moe |
other — other (4) | 4 | #50145 | 🔒 task |
qwen3_vl_moe |
OOM — other (3) | 3 | #50148 | 🔒 task |
colqwen2 |
other — other (2) | 2 | 🔒 task | |
deepseek_vl_hybrid |
OOM — other (2) | 2 | #49581 | 🔒 task |
deepseek_vl_hybrid |
cuda_runtime — other (1) | 1 | #49335 | 🔒 task |
Not dispatched — environment / dependency
These groups were triaged but NOT handed to Serge: their failure mode is a property of the runner or the environment, so no minimal source patch can fix them. They need a human (runner capacity, a dependency pin).
1 model ran out of device memory (3 failures) — needs runner capacity, not a patch, so none of these were dispatched: llama4 (3).
6 group(s) skipped — stopped failing before this run (grounding_dino, mm_grounding_dino, oneformer, llava_next_video +2 more). Not dispatched and not awaiting a human.
Outcome recap
Why each group that opened no PR ended without one, and what it cost — surfaced here from the Serge dashboard. Failing tests links straight to each test's dashboard view, since these are the groups a human has to pick up.
| Group | Reason | LLM | Tokens (in / out) | Failing tests |
|---|---|---|---|---|
pocket_tts (regressed by PR #49589) |
[reproduced] GPU reproduce + classify: this is an ENVIRONMENT issue (All 16 failures are missing external artifacts — the nonexistent eustlb/pocket-tts-english-drifting Hub repo, the absent en-Alice_woman.wav in the pinned dummy-audio-samples snapshot, and the missing processing/metadata.json bucket file (with the lone output mismatch likely from the metadata's recorded torch 2.14 vs CI's 2.15) — none of which a source patch can supply.) — no source patch can fix it, so no investigation and no PR. |
deepseek-ai/DeepSeek-V4.1-Flash-1017:together |
— / — | test_drifting · test_training · test_voice_cloning · +5 more |
vipllava (regressed by PR #49594) |
The repository's normalizer (`bash -lc — normalizer: 1 failed: modeling_rules_doc | deepseek-ai/DeepSeek-V4.1-Flash-1017:together |
2,125,687 / 100,877 | test_small_model_integration_test |
diffusion_gemma |
No patch: the reproduced diffs are not expectation drift. | deepseek-ai/DeepSeek-V4.1-Flash-1017:together |
2,176,789 / 81,084 | test_diffusion_gemma_chat_template · test_minified_diffusion_gemma_forward_batched · test_minified_diffusion_gemma_forward_text_only · +1 more |
blt |
The repository's normalizer (`bash -lc — normalizer: 1 failed: modeling_rules_doc | deepseek-ai/DeepSeek-V4.1-Flash-1017:together |
2,141,528 / 54,890 | test_model_logits · test_model_logits_bf16 |
colqwen2 |
The repository's normalizer (`bash -lc — normalizer: 1 failed: modeling_rules_doc | deepseek-ai/DeepSeek-V4.1-Flash-1017:together |
2,053,852 / 89,765 | test_model_integration_test |
`vipllava` (regressed by PR #49594) — normalizer output (tail)
--- omitted 3511 leading chars ---
registry(rule_specs_path)
File "/usr/local/lib/python3.11/site-packages/mlinter/mlinter.py", line 511, in _activate_rule_registry
"TRF_RULE_CHECKS": _build_rule_checks(rule_specs, deprecated_rules),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mlinter/mlinter.py", line 443, in _build_rule_checks
raise ValueError(f"Rule {rule_id} is marked deprecated but {module_name}.py still exists; delete it.")
ValueError: Rule TRF001 is marked deprecated but trf001.py still exists; delete it.
FAILED (0.27s)
Docstring formatting
$ python utils/check_docstrings.py --fix_and_overwrite
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
OK (2.32s)
Dummy objects
$ python utils/check_dummies.py --fix_and_overwrite
OK (0.06s)
Pipeline type hints
$ python utils/check_pipeline_typing.py --fix_and_overwrite
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
OK (1.97s)
Doctest list
$ python utils/check_doctest_list.py --fix_and_overwrite
OK (0.05s)
Model dates
$ python utils/add_dates.py
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
OK (1.34s)
Dependency versions table
$ python setup.py deps_table_update
running deps_table_update
OK (0.32s)
1 failed: modeling_rules_doc
Using Python 3.11.18 environment at: /usr/local
Resolved 1 package in 1.72s
Building transformers @ file:///tmp/serge-clones/worktrees/huggingface__transformers__task__main__ebee0dffd22c4a18a57d91da9561b856
Built transformers @ file:///tmp/serge-clones/worktrees/huggingface__transformers__task__main__ebee0dffd22c4a18a57d91da9561b856
Prepared 1 package in 3.99s
Uninstalled 1 package in 0.92ms
warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.
If the cache and target directories are on different filesystems, hardlinking may not be supported.
If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.
Installed 1 package in 4ms
~ transformers==5.24.0.dev0 (from file:///tmp/serge-clones/worktrees/huggingface__transformers__task__main__ebee0dffd22c4a18a57d91da9561b856)
make: *** [Makefile:63: fix-repo] Error 1
`blt` — normalizer output (tail)
--- omitted 3511 leading chars ---
registry(rule_specs_path)
File "/usr/local/lib/python3.11/site-packages/mlinter/mlinter.py", line 511, in _activate_rule_registry
"TRF_RULE_CHECKS": _build_rule_checks(rule_specs, deprecated_rules),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mlinter/mlinter.py", line 443, in _build_rule_checks
raise ValueError(f"Rule {rule_id} is marked deprecated but {module_name}.py still exists; delete it.")
ValueError: Rule TRF001 is marked deprecated but trf001.py still exists; delete it.
FAILED (0.24s)
Docstring formatting
$ python utils/check_docstrings.py --fix_and_overwrite
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
OK (2.44s)
Dummy objects
$ python utils/check_dummies.py --fix_and_overwrite
OK (0.05s)
Pipeline type hints
$ python utils/check_pipeline_typing.py --fix_and_overwrite
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
OK (2.04s)
Doctest list
$ python utils/check_doctest_list.py --fix_and_overwrite
OK (0.06s)
Model dates
$ python utils/add_dates.py
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
OK (1.32s)
Dependency versions table
$ python setup.py deps_table_update
running deps_table_update
OK (0.33s)
1 failed: modeling_rules_doc
Using Python 3.11.18 environment at: /usr/local
Resolved 1 package in 1.53s
Building transformers @ file:///tmp/serge-clones/worktrees/huggingface__transformers__task__main__204f46d9a98c45a9a50afb17a9585f97
Built transformers @ file:///tmp/serge-clones/worktrees/huggingface__transformers__task__main__204f46d9a98c45a9a50afb17a9585f97
Prepared 1 package in 4.12s
Uninstalled 1 package in 0.98ms
warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.
If the cache and target directories are on different filesystems, hardlinking may not be supported.
If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.
Installed 1 package in 4ms
~ transformers==5.24.0.dev0 (from file:///tmp/serge-clones/worktrees/huggingface__transformers__task__main__204f46d9a98c45a9a50afb17a9585f97)
make: *** [Makefile:63: fix-repo] Error 1
`colqwen2` — normalizer output (tail)
--- omitted 3508 leading chars ---
le_registry(rule_specs_path)
File "/usr/local/lib/python3.11/site-packages/mlinter/mlinter.py", line 511, in _activate_rule_registry
"TRF_RULE_CHECKS": _build_rule_checks(rule_specs, deprecated_rules),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mlinter/mlinter.py", line 443, in _build_rule_checks
raise ValueError(f"Rule {rule_id} is marked deprecated but {module_name}.py still exists; delete it.")
ValueError: Rule TRF001 is marked deprecated but trf001.py still exists; delete it.
FAILED (0.24s)
Docstring formatting
$ python utils/check_docstrings.py --fix_and_overwrite
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
OK (2.47s)
Dummy objects
$ python utils/check_dummies.py --fix_and_overwrite
OK (0.10s)
Pipeline type hints
$ python utils/check_pipeline_typing.py --fix_and_overwrite
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
OK (2.15s)
Doctest list
$ python utils/check_doctest_list.py --fix_and_overwrite
OK (0.05s)
Model dates
$ python utils/add_dates.py
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
OK (1.32s)
Dependency versions table
$ python setup.py deps_table_update
running deps_table_update
OK (0.33s)
1 failed: modeling_rules_doc
Using Python 3.11.18 environment at: /usr/local
Resolved 1 package in 1.54s
Building transformers @ file:///tmp/serge-clones/worktrees/huggingface__transformers__task__main__459ab64801484a51aa2c651d888e8953
Built transformers @ file:///tmp/serge-clones/worktrees/huggingface__transformers__task__main__459ab64801484a51aa2c651d888e8953
Prepared 1 package in 4.10s
Uninstalled 1 package in 1ms
warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.
If the cache and target directories are on different filesystems, hardlinking may not be supported.
If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.
Installed 1 package in 5ms
~ transformers==5.24.0.dev0 (from file:///tmp/serge-clones/worktrees/huggingface__transformers__task__main__459ab64801484a51aa2c651d888e8953)
make: *** [Makefile:63: fix-repo] Error 1
Generated 2026-11-13T23:46:00+00:00.
Metadata
Metadata
Assignees
Labels
No labels
Activity