Repository navigation
Commit cd8cc45
authored
feat(ci): surface pre-existing failures in PR run-slow comment (#50122)
* feat(ci): surface pre-existing failures in PR CI comment
When a PR fixes a pre-existing test failure, `run-slow` previously
silently dropped it (failing on both PR and base = "not new"). This
made it impossible to verify the fix via the CI comment.
The report now buckets failures into three categories:
- New (✅ ⟹ ❌): bad_commit is not None, no "DIFFERENT error message"
- Error changed (❌ ⟹ ❌): bad_commit is not None, "DIFFERENT error message"
- Same error (❌ ⟹ ❌): bad_commit is None, failure_at_base_commit non-empty
Flaky / bisect-failed entries (bad_commit=None, failure_at_base_commit='')
are still silently discarded as before.
The PR commit status (green/red) is based on new failures only, so a PR
with only pre-existing failures still gets a green check. Pre-existing
failures are shown below the status line in an always-visible section.
Same-error lists with >10 entries are collapsed behind a "show all N"
toggle using the dl/dd trick to avoid ugly blockquote styling.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(ci): keep error_changed failures red, fix unused vars
- error_changed (bad_commit set, different error message) was accidentally
moved to green by the previous commit. The gate now fires (STATUS=failure)
when new_count + error_changed_count > 0, matching the old behaviour where
bad_commit is not None counted as a failure.
- When only error_changed failures are present (no new), the header says
"N tests with changed errors" instead of "N new failing tests".
- Unused preexisting_count return values replaced with _.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor(ci): extract report formatting into utils/format_ci_report.py
Move the inline Python logic from self-comment-ci.yml and self-comment-ci-amd.yml
into a shared script so both workflows stay in sync. The script accepts
--report-path, --url-path, --header-out, --details-out, and --kind (model|quantization).
This also brings self-comment-ci-amd.yml up to date with the new bucketing
logic (pre-existing failures now visible in the AMD section of the PR comment).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(ci): show same-error details in AMD success path
Mirror the detail-showing logic from the Nvidia workflow: when
STATUS=success but model_ci_details.txt exists (same-error-only
failures), cat it into the comment body so the AMD section is
consistent with the Nvidia section.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: ruff format and fix PERF102 in format_ci_report.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(ci): add checkout to report job so format_ci_report.py is available
The report job had no checkout step, so python3 utils/format_ci_report.py
would fail with "can't open file" — the workspace only contains downloaded
artifacts, not the repo. Add actions/checkout and contents: read permission
to both self-comment-ci.yml and self-comment-ci-amd.yml.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(ci): use fetch-depth=1 in report job checkout
We only need utils/format_ci_report.py, not the full git history.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(ci): group by (model, job_link) not model alone in format_entries_by_model
A model failing on both single-gpu and multi-gpu has different job_links
per device. Grouping only by model meant all tests were listed under the
first device's link. Group by (model, job_link) to emit a separate header
per device, matching the old behaviour.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(ci): improve header label and remove fragile status string check
- Header now shows both parts when new and error_changed coexist:
e.g. "❌ [2 new failing tests · 1 tests with changed errors](url) 😭"
so the one-liner accurately reflects all red-gate failures.
- Remove redundant "SAME error message" in status check: bad_commit is None
is already implied by the elif branch, and failure_at_base non-empty
uniquely identifies same-error entries per check_bad_commit.py logic.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* docs(ci): add TODO for potential 65536-char GitHub comment limit
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>1 parent aaf32f8 commit cd8cc45
3 files changed
Lines changed: 269 additions & 194 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
277 | 277 | | |
278 | 278 | | |
279 | 279 | | |
| 280 | + | |
280 | 281 | | |
281 | 282 | | |
282 | 283 | | |
283 | 284 | | |
284 | 285 | | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
285 | 291 | | |
286 | 292 | | |
287 | 293 | | |
| |||
309 | 315 | | |
310 | 316 | | |
311 | 317 | | |
312 | | - | |
313 | | - | |
314 | | - | |
315 | | - | |
316 | | - | |
317 | | - | |
318 | | - | |
319 | | - | |
320 | | - | |
321 | | - | |
322 | | - | |
323 | | - | |
324 | | - | |
325 | | - | |
326 | | - | |
327 | | - | |
328 | | - | |
329 | | - | |
330 | | - | |
331 | | - | |
332 | | - | |
333 | | - | |
334 | | - | |
335 | | - | |
336 | | - | |
337 | | - | |
338 | | - | |
339 | | - | |
340 | | - | |
341 | | - | |
342 | | - | |
343 | | - | |
344 | | - | |
345 | | - | |
346 | | - | |
347 | | - | |
348 | | - | |
349 | | - | |
350 | | - | |
351 | | - | |
352 | | - | |
353 | | - | |
354 | | - | |
355 | | - | |
356 | | - | |
357 | | - | |
358 | | - | |
359 | | - | |
360 | | - | |
361 | | - | |
362 | | - | |
363 | | - | |
364 | | - | |
365 | | - | |
366 | | - | |
367 | | - | |
368 | | - | |
369 | | - | |
370 | | - | |
371 | | - | |
372 | | - | |
373 | | - | |
374 | | - | |
375 | | - | |
376 | | - | |
377 | | - | |
378 | | - | |
379 | | - | |
380 | | - | |
381 | | - | |
382 | | - | |
383 | | - | |
384 | | - | |
385 | | - | |
386 | | - | |
387 | | - | |
388 | | - | |
389 | | - | |
390 | | - | |
391 | | - | |
392 | | - | |
393 | | - | |
394 | | - | |
395 | | - | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
396 | 324 | | |
397 | 325 | | |
398 | 326 | | |
| |||
433 | 361 | | |
434 | 362 | | |
435 | 363 | | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
436 | 368 | | |
437 | 369 | | |
438 | 370 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
265 | 265 | | |
266 | 266 | | |
267 | 267 | | |
| 268 | + | |
268 | 269 | | |
269 | 270 | | |
270 | 271 | | |
271 | 272 | | |
272 | 273 | | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
273 | 279 | | |
274 | 280 | | |
275 | 281 | | |
| |||
312 | 318 | | |
313 | 319 | | |
314 | 320 | | |
315 | | - | |
316 | | - | |
317 | | - | |
318 | | - | |
319 | | - | |
320 | | - | |
321 | | - | |
322 | | - | |
323 | | - | |
324 | | - | |
325 | | - | |
326 | | - | |
327 | | - | |
328 | | - | |
329 | | - | |
330 | | - | |
331 | | - | |
332 | | - | |
333 | | - | |
334 | | - | |
335 | | - | |
336 | | - | |
337 | | - | |
338 | | - | |
339 | | - | |
340 | | - | |
341 | | - | |
342 | | - | |
343 | | - | |
344 | | - | |
345 | | - | |
346 | | - | |
347 | | - | |
348 | | - | |
349 | | - | |
350 | | - | |
351 | | - | |
352 | | - | |
353 | | - | |
354 | | - | |
355 | | - | |
356 | | - | |
357 | | - | |
358 | | - | |
359 | | - | |
360 | | - | |
361 | | - | |
362 | | - | |
363 | | - | |
364 | | - | |
365 | | - | |
366 | | - | |
367 | | - | |
368 | | - | |
369 | | - | |
370 | | - | |
371 | | - | |
372 | | - | |
373 | | - | |
374 | | - | |
375 | | - | |
376 | | - | |
377 | | - | |
378 | | - | |
379 | | - | |
380 | | - | |
381 | | - | |
382 | | - | |
383 | | - | |
384 | | - | |
385 | | - | |
386 | | - | |
387 | | - | |
388 | | - | |
389 | | - | |
390 | | - | |
391 | | - | |
392 | | - | |
393 | | - | |
394 | | - | |
395 | | - | |
396 | | - | |
397 | | - | |
398 | | - | |
399 | | - | |
400 | | - | |
401 | | - | |
402 | | - | |
403 | | - | |
404 | | - | |
405 | | - | |
406 | | - | |
407 | | - | |
408 | | - | |
409 | | - | |
410 | | - | |
411 | | - | |
412 | | - | |
413 | | - | |
414 | | - | |
415 | | - | |
416 | | - | |
417 | | - | |
418 | | - | |
419 | | - | |
420 | | - | |
421 | | - | |
422 | | - | |
423 | | - | |
424 | | - | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
425 | 333 | | |
426 | 334 | | |
427 | 335 | | |
| |||
481 | 389 | | |
482 | 390 | | |
483 | 391 | | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
484 | 400 | | |
485 | 401 | | |
486 | 402 | | |
| |||
0 commit comments