🩺 CI Health — Last 7 days

[radical]

_{Window: last 7d (Aug 10–16, 2026) · vs prior 7d (Aug 3–9) · trend: last 18 days · generated 2026-08-16 · numbers are volume-weighted pooled counts (a low-build weekend day can't swing them — see note ▾)}

📊 Are we improving?

Net: improving. The flagship 🔀 PR lane recovered to 79% (414/521, ▲18 WoW on 521 real runs) as last week's Aug 8–9 mass-failure event rolled into the prior window — this week ran healthy (95% Aug 11, 85% Aug 13, 79% Aug 14, 59% partial Aug 16). 🏗️ internal main held (80%, ▲1). The one drag is rolling red: 🌳 GH main fell to 27% (21/78, ▼29), red-heavy all week — but it's a low-volume no-rerun-net lane. Weighted toward the steady high-volume PR lane, the week is genuinely better; GH-main rolling red is the standing thing to watch.

	Lane pass (same lanes & order as §🚦)	prior 7d → this 7d	Δ
↓	🌳 GH main (rolling) pass	56% → 27%	▼29 (prior 28/50; this 21/78; rolling has no rerun net — swingy, red-heavy)
↓	🌿 GH release/13.5 push pass	50% → 46%	▼4 (tiny n: 6/13; prior 1/2)
→	🧪 GH outerloop main pass	29% → 29%	→0 (tiny n: 2/7; most recent scheduled run passed)
↑	🏗️ Internal AzDO main pass	79% → 80%	▲1 (prior 19/24; this 33/41, 8 fails — holding)
↑	🏗️ Internal release/13.5 pass	0% → 69%	new (tiny n: 9/13; prior 0/1)
↑	🔀 PR validation pass	61% → 79%	▲18 (real volume: 414/521; prior window carried the Aug 8–9 collapse, now aged out)
⚪	broken backlog	ciinsights blind since ~Jun 30 (~47d) — can't confirm	last-known retained slice (≤ Jun 30): CLI-E2E + conformance + Hosting-provider families

🚦 Pass rate by lane

Lane	Pass (✅/total)	Δ vs prior 7d
🌳 GH CI — main	🔴 27% (21/78)	▼ 29 pts (prior 28/50; rolling, no rerun net — swingy but red-heavy)
🌿 GH CI — release/13.5	🟡 46% (6/13)	▼ 4 pts (tiny n)
🧪 GH CI outerloop — main	🔴 29% (2/7)	→ 0 pts (tiny n; most recent scheduled run passed)
🏗️ Internal (AzDO) — main	🟢 80% (33/41)	▲ 1 pt (prior 19/24; 8 fails, latest failed build)
🏗️ Internal (AzDO) — release/13.5	🟡 69% (9/13)	new (tiny n; latest failed Aug 14)
🔀 GH CI — PR validation ⭐	🟢 79% (414/521)	▲ 18 pts (real volume; healthy all week, 85% Aug 13)

_{Outerloop-release omitted — no scheduled runs on release/13.5.}

🧱 infra share: n/a fresh this window — ciinsights ingested no Jul/Aug builds (newest ~Jun 30), so an in-window infra/test split can't be computed. Last-known standing pattern (retained slice ≤ Jun 30): ~74% of failed builds were environmental timeout-style infra, not product regressions. Details ▾.

📈 Trend — pass rate over the last ~18 days

_{Three lanes on the same direct gh-run/AzDO basis. Low-build days are drawn hollow/faded and marker size ∝ #builds, so a 1–2 build day doesn't read as a real swing. Aug 16 (rightmost, boxed) is a partial day — ignore its position. This tier recomputes only the day it runs and freezes prior published days, so Aug 02–09 have no per-day rows (no weekly run landed on them); the trend picks up again at Aug 10. 🔵 PR validation recovered (79% pooled; 85% Aug 13, 79% Aug 14). 🟢 GH main swings on few builds/day and is red-heavy this week (27% pooled; 0/7 Aug 13). 🟠 internal AzDO main held (80%; the Aug 14 dip is a 1/2 tiny-n day).}

📅 Per-day numbers — PR, GH main, internal AzDO main (newest first)

Day	🔀 PR validation	🌳 GH main	🏗️ Internal main
Aug 16	59% (10/17)	—	—
Aug 15	75% (9/12)	0% (0/3)	—
Aug 14	79% (15/19)	33% (1/3)	50% (1/2)
Aug 13	85% (22/26)	0% (0/7)	80% (4/5)
Aug 12	71% (27/38)	15% (2/13)	100% (8/8)
Aug 11	95% (71/75)	25% (1/4)	67% (2/3)
Aug 10	59% (74/126)	75% (3/4)	100% (3/3)
Aug 01	86% (6/7)	—	—
Jul 31	92% (11/12)	—	—
Jul 30	73% (11/15)	100% (2/2)	—
Jul 29	86% (6/7)	0% (0/1)	100% (1/1)
Jul 27	65% (11/17)	40% (2/5)	100% (4/4)
Jul 24	90% (9/10)	50% (1/2)	100% (1/1)
Jul 22	90% (19/21)	0% (0/1)	100% (1/1)
Jul 21	93% (14/15)	100% (3/3)	100% (3/3)
Jul 20	82% (14/17)	100% (2/2)	0% (0/1)
Jul 17	67% (4/6)	100% (2/2)	0% (0/1)
Jul 16	100% (3/3)	—	—

_{Direct counts (completed runs, success/(success+failure)); internal counts "partially succeeded" as success. Italic = low build count that day — read directionally. Today's row (Aug 16) recomputed this run; earlier PR/main/internal cells frozen verbatim (first-computation snapshots, so a frozen cell like Aug 11 PR 71/75 — a partial snapshot from an earlier run — won't match its fuller fresh-fetch count and need not sum to the §3 pooled headline). Aug 02–09 have no rows (no weekly run landed on them). Oldest day rolls off at ~18 rows.}

⏱️ Build times — by pipeline

p50/p95 over the same completed-run set as pass rate; PR and push cohorts kept separate (never averaged).

Pipeline	p50 (this wk)	p95 (this wk)	Δ p50 vs prior wk
🌳 GH CI — main (push)	50m	80m	▲ 15m (was 35m)
🌿 GH CI — release/13.5 (push)	31m	53m	▼ 11m (was 42m, prior n=2)
🏗️ Internal (AzDO) — main	83m	124m	▲ 4m (was 79m)
🏗️ Internal (AzDO) — release/13.5	80m	154m	new (n=13; prior n=1)

_{GH durations: run_started_at→updated_at. Internal: startTime→finishTime. GH main p50 is 50m (▲15m WoW) — tracking the red-heavy main runs this week (red runs that ran the full matrix before failing are slower than a clean pass); p95 a tame 80m. Internal main p50 flat (79m→83m), p95 steady at 124m. release/13.5 rows have thin/absent prior baselines (2 GH-push, 1 internal last week) — read the Δ directionally. PR validation build time is intentionally excluded: ci.yml on pull_request is only the short gating workflow — the real PR test work runs in separately-triggered workflows it doesn't wait on — so its run duration is not end-to-end PR CI time and there is no cheap accurate figure. (PR-validation pass rate still appears in §2/§3.)}

🔧 Triage worklist — fix these first

⚠️ This entire list is STALE and NOT confirmable this run — ciinsights's newest ingested test failure is ~Jun 30 (~47 days; the whole Jul 1–Aug 16 window unseen). A 7d / 30d health-summary returns 0 builds (get_flagged_tests → no_builds); a ~75d query times out. The families below are the last retained slice (all last-failed Jun 28–30); ciinsights cannot tell us whether they are fixed, stopped running, or simply un-ingested. Do not treat any row as live work — every row is ≥47 days stale, so all need "confirm fixed vs stopped running." The reliable signal this week is the lane pass rates (§2/§3), sourced from gh/az directly: PR 79% (healthy all week, 85% Aug 13), GH-main 27% (red-heavy), outerloop 2/7 (most recent run passed), internal 80%.

Severity	fails (retained, ≤ Jun 30)	Test	Last failed
🔴 broken (stale?)	~1300+ fails (100%), 58 tests	Aspire.Cli.EndToEnd / Tests harness — Ollama, OtelLogs, Describe/ResourceCommand, LogsCommand, AgentMcp, Smoke LatestCli, StopNonInteractive, EmptyAppHostTemplate…; all fail the CLI E2E automation harness = shared harness root cause. Freshest family (Jun 30).	Jun 30 ⚠️ ~47d — confirm fixed vs stopped
🔴 broken (stale?)	~180 fails (100%), 12 tests	Conformance family across EFCore.SqlServer, Qdrant, RabbitMQ, MongoDB and other providers — LoggerFactoryIsUsedByRegisteredClient, ServiceLifetimeIsAsExpected, Tracing/HealthChecksRegisters…; same conformance-suite shape across providers = likely one shared fix, not per-provider.	Jun 29 ⚠️ ~48d — confirm fixed vs stopped
🔴 broken (stale?)	~2000+ fails (100%), ~90 tests	Aspire.Hosting. provider integration family* — MongoDB, Kubernetes/Helm, PostgreSQL, JavaScript, RabbitMQ, SqlServer, Python, MySql, Nats, Redis, Containers… broad container/integration outage across providers. Older edge (last Jun 28) — most likely already off if ingestion resumes.	Jun 28 ⚠️ ~49d — confirm fixed vs stopped
🟡 flaky	—	None flagged — get_flagged_tests returned 0 flaky / 0 broken with no_builds (7d window empty). The live flake proxy is the rerun volume (§9), which stayed elevated (469 bot reruns → 127 PRs rescued).	n/a

_{Counts are per-test failure totals over ciinsights' retained slice (all pre-window, ≤ Jun 30) on the query_test_failures 100%-rate basis. Rows group same-error families so one root cause is one row (fails summed across the grouped tests; family totals ~approximate as the slice is capped at the fetched page). Everything is 47–49d stale — treat as leads to confirm (fixed vs stopped running), not live work. Distinct root causes → triage separately.}

---

📖 Dominant story & week-over-week detail — flagship PR lane recovered (79%, ▲18) as the Aug 8–9 collapse aged out; GH-main rolling red (27%, ▼29); ciinsights blind ~47 days

This week's dominant story is the PR lane's recovery — 61%→79% (▲18) on 521 runs. Most of that lift is the Aug 8–9 mass-failure event rolling out of the this-7d window (it now sits in the prior window and drags its 61%), but the this-week runs are genuinely healthy on their own terms: 95% Aug 11, 85% Aug 13, 79% Aug 14, 59% partial Aug 16. Weighted toward the highest-volume flagship lane, that recovery is what pins the verdict at improving rather than merely holding.

The thing that moved the wrong way is 🌳 GH main — 56%→27% (▼29). It was red-heavy all week (0/7 Aug 13) and swings on ~11 builds/day, so the pooled figure overstates a clean trend, but 27% over 78 runs is a genuinely weak rolling week, not pure noise — and rolling main has no rerun net, so a human has to notice. This is the standing action item now that the PR incident is behind us.

🧪 Outerloop main stayed thin and mostly red (2/7, flat WoW) but its most recent scheduled run passed. One green isn't a recovery, but it's a change of direction worth watching in the daily Pulse (#18232). 🏗️ Internal main held flat and healthy (79%→80%); its lone Aug 14 dip is a 1/2 tiny-n day, not a trend.

The flaky tax stayed elevated even as the lane recovered. Reruns rescued a large batch of PRs — 🤖 the auto-rerun bot fired 469 rerun attempts across 289 PR runs → 127 distinct PR branches rescued red→green, with 12 branches never going green despite reruns (see §9). Heavy rerun volume with a recovered pooled pass rate means the PR lane is absorbing normal-week flake/transient-infra automatically, not masking a mass regression this time.

ciinsights's newest ingested test failure is ~Jun 30 — ~47 days, spanning several weekly reports. A 7/30-day health-summary returns 0 builds (no_builds); a ~75d query times out. Every "broken" family (§🔧) is that retained slice and the entire Jul–Aug window is unseen; treat the backlog as unconfirmed, not cleared. §3 lane table and §4 per-day trend share one direct gh/az basis (they agree on method). With ciinsights blind, the rerun volume (§9) is the only live proxy for how much flake/transient-infra the PR lane is absorbing.

⚖️ Why a low-build weekend can't mislead the headline (weighting note)

The lane %s and WoW deltas are pooled counts — Σsuccess / Σtotal over the 7-day window — not an average of daily rates. So a quiet day contributes only its raw builds to a 521-build PR denominator, not a full "day." A quiet week can't swing a pooled figure the way it swings a single day's ratio.

Where low volume does distort is the per-day chart/grid: there a 1–2 build day reads as a full 0%/100% point. Handled visually — low-build days hollow/faded with marker size ∝ #builds, today's partial day boxed, low-build cells italicized. A faded point = "not enough signal," not a real swing. (This is why the release rows and the thin internal-main day cells are called tiny-n.)

(Considered but deferred: a 7-day trailing volume-weighted overlay line. More rigorous but adds a second line per lane and hurts readability; the pooled headline + low-n de-emphasis already remove the misleading-spike problem.)

🔁 Auto-rerun & the flaky tax — reruns rescued 134 distinct PR branches this week; 12 never went green

PR-validation reruns (run_attempt > 1, 7d):

• 🤖 Auto-rerun bot: 469 rerun attempts across 289 PR runs → 127 distinct PRs rescued red→green; 13 still red.
• 🧑 Human reruns: 97 rerun attempts across 41 PR runs → 17 distinct PRs rescued; 16 still red.
• 🔗 Combined: 146 distinct PR branches needed ≥1 rerun; 134 went green, 12 never went green (27 branches needed both bot + human).

Rerun volume stayed high even as the lane recovered — the flaky/transient tax the PR lane pays every week, mostly absorbed automatically. The 12 branches that never went green despite reruns are the residue of deterministic (non-flaky) failures that reruns can't fix. With ciinsights blind, this rerun volume is currently the only live proxy for how much flake/transient-infra the PR lane is absorbing. (Auto-rerun is PR-only; rolling main/release/outerloop-scheduled reds above are raw, un-rescued. Attribution reflects each run's latest attempt only; distinct-PR counts dedupe on head branch.)

🧱 Infrastructure failures — fresh share not computable (ciinsights blind, newest ~Jun 30); last-known ~74%

A fresh in-window infra share can't be computed this run — ciinsights, the source that classifies a failed build as infra vs test, has ingested no Jul/Aug builds (newest ~Jun 30) and returns 0 in-window builds. The standing pattern from its retained slice (≤ Jun 30) is unchanged: the majority of failed builds are environmental — long-duration timeout-style infra on genuine branches, mostly rescued on rerun, inflating the failed-build denominator but rarely blocking merge. Last-known figure: ~74% of failed builds over the retained set. That denominator is attempt-level and ends ~Jun 30, so do not read it as this week's rate. Group by base branch + job name before acting on any headline %. Once ingestion catches up, recompute the in-window share. (Source: ciinsights build-health failure breakdown over its rolling build set — a different, and currently stale, denominator than the per-lane counts above.)

_{Sources: ciinsights MCP/API (test/infra — blind this run: DB connected but no Jul/Aug builds ingested; newest test failure ~Jun 30, so 7/30d health-summaries return 0 builds / no_builds, a ~75d query times out; get_flagged_tests returned 0 flaky / 0 broken this run; last-known figures are from that retained ≤ Jun 30 slice, all on direct basis) + gh api .../ci.yml/runs (lane pass rates, rerun outcomes — complete, PR fetched per-date-bucketed and verified against each bucket to beat the 1000-run cap) + az pipelines (internal def 1602; counts only — complete). Lane %s are volume-weighted pooled counts. Lane table and per-day trend on one direct basis; PR/main/internal cells frozen pre-today (internal-main backfilled once then frozen). Retention ≈18d. Window = trailing 7d (Aug 10–16). Read-only report — no tests changed. What's on fire right now: see the daily CI Pulse (#18232). Generated 2026-08-16 19:16 UTC.}

[joperezr]

🤖 Aspire triage pass — 2026-06-16 06:54 UTC

Area: �rea-engineering-systems
Kind: docs (informational CI health report — weekly pulse)
Repro: n/a — generated dashboard, not a bug report
Affected version(s): n/a (CI health snapshot, window: 6d ending 2026-06-16)
Severity: S4 — informational; no product regression. Body surfaces real broken/flaky tests (SqlServer conformance family, Aspire.Templates.Tests.*.CanNewAndBuild, outerloop on main red 6/6) that should be tracked as separate failing-test issues if not already.

Urgency: ⚪ backlog — no milestone
Reason: informational weekly dashboard, not a fix candidate. The triage worklist it contains is for human action via existing failing-test issue process.

Suggested owner(s): Jose Perez Rodriguez (@joperezr) (score 6)
Evidence:

• joperezr merged Merge release/13.4 to main after v13.4.0 release #17817, Merge release/13.3 to main after v13.3.5 release (supersedes #17337) #17343 in area-engineering-systems (last 90d)
• Issue author (Ankit Jain (@radical)) is excluded from owner suggestions per policy

Confidence: medium
Needs human?: no — informational. Worklist items inside (SqlServer conformance cluster, template CanNewAndBuild failures) should be tracked under their own failing-test issues, not this rollup.

Dry-run mode — no labels or milestone applied. Reply /triage apply to promote, or edit and reapply.

[radical]

🔧 Post-run review — 2026-06-18 19:57 UTC

Regressions vs previous run: none mechanical — frozen per-day rows (Jun 01–17) are byte-identical, the Jun 18 row and the trend PNG were preserved verbatim. The visible content changes (headline number, lead story) are deliberate, see below.

Biggest issue this run — dual-basis number mismatch (worth a human decision):

• The lane table is computed per the prompt's explicit spec (gh run list --workflow ci.yml --event pull_request etc.) → PR validation = 76% (262/343), with prior-6d on the same basis = 45%, a real +31 pts (Jun 16–18 direct = 84/79/85%).
• The per-day trend (and the prior run's headline of 33% (219/670)) is on the older differenced/cumulative basis, which corresponds to ciinsights buildHealth (all lanes lumped, 737 builds @ 33%). Its per-day PR denominators (134, 178, 146…) exceed the actual total ci.yml pull_request run count for that day, so it is not the same population as the lane query the prompt mandates.
• Net: this run's lane table says PR ~76% while the trend chart/table says ~30–41%. I bridged with an explicit ⚠️ basis note and kept the trend frozen, but the two sections genuinely disagree on level.
• Suggested prompt change (human to decide): in aspire-ci-health-weekly.prompt.txt, make the §2 per-day trend use the same direct per-lane gh-run counting as the §1 lane table (and regenerate the PNG from those), so headline and trend reconcile. Today the freeze rule is preserving a series that is ~2× low on PR. Alternatively, state in the prompt that the lane table and the trend are intentionally different populations (lane = per-lane gh runs; trend = lumped buildHealth) so future runs stop trying to reconcile them.

Other deliberate content changes (real state, not regressions):

• Lead story moved from "outerloop flipped red" to "EF Core SqlServer conformance+enrich suite escalated flaky→broken (100%)" — ciinsights get_flagged_tests now lists the whole suite at 100% failure (LoggerFactory… 60/60 + ~34 siblings at 30/30 and 15/15); last week it was the top flaky cluster. This is the correct new headline.
• Headline PR figure changed 33%→76% (basis correction above), with a same-method +31 pt delta vs prior 6d.

Data-gathering issues this run:

• GH Actions API 1000-result cap: an unbounded created>= fetch of pull_request runs truncated at 1000 (only reached Jun 07). Worked around by fetching each 6d window separately (created=2026-06-12..2026-06-18, …06-06..06-11), each <1000. The unbounded fetch silently undercounts the oldest slice — future runs should always window-bound the PR fetch.
• ciinsights DB warning: connection healthy but lastError reported a duplicate-index (IX_TestResults_FullyQualifiedName_CapturedAt); queries still returned data.
• ciinsights endpoint disagreement: get_failure_summary.topFlaky lists EntityFrameworkCore.SqlServer…LoggerFactory as flaky 36/84 (42%), while get_flagged_tests lists the same test as broken 60/60 (100%). Different windows/dedup. I trusted the broken (100%) detector for the escalation call and kept the residual Metrics/HealthCheck cluster as the flaky row.

Same-day re-run note: this was the 2nd weekly run today (prior at 19:06 UTC). I kept the entire per-day series — including the Jun 18 row — frozen, rather than recompute today on the trend's (incompatible) basis. If the intent is that a same-day re-run should recompute today's row, the prompt should say which basis to use; recomputing it on the lane basis (PR 85%) would make today incomparable to every frozen row.

[radical]

🔧 Post-run review — 2026-06-19 19:18 UTC

Regressions vs previous run: none. All six sections preserved; the Jun 02–18 per-day trend cells are byte-identical to the prior body (only the (today) tag moved off Jun 18); Jun 01 correctly aged out to keep the 18-day window; title left untouched (body-only update).

Readability / improvement ideas:

• The per-day-trend basis split is now a recurring, load-bearing defect — please resolve it (3rd run carrying the ⚠️ note). Because Jun 02–18 are frozen on the old differenced/cumulative basis (~20–46%) while today's row is computed on the direct gh-run basis (Jun 19 PR = 87%), the trend chart shows a ~46pt step-up at the right edge that looks like a real one-day event but is a pure basis artifact. The lane table (PR 81% direct) is the truth. Two clean fixes, pick one decisively: (a) migrate the whole series to direct completed-count basis with a one-time history reset (best — the differenced basis is acknowledged-wrong and understates by ~2×), or (b) keep computing today's row on the same differenced basis as history so the line stays continuous. The current "freeze history + direct today" hybrid produces a misleading chart every run. Accept/reject: a human should decide (a) vs (b); I did not unilaterally reset frozen history this run.
• Consider dropping the per-day denominators that are on the broken basis (e.g. 41% (55/134)) until the basis is reconciled — the parenthetical counts invite readers to trust a level the prompt explicitly says is wrong.

Data-gathering issues this run:

• GH runs API --paginate hard-caps near 1000 results. A single event=pull_request&created=>=2026-06-05 paginate stopped at Jun 09, silently truncating the prior-6d (Jun 06–08) PR slice. I had to issue a second date-bounded fetch (created=2026-06-06..2026-06-12) to compute the prior-window delta. This will bite every prior-vs-current PR comparison.
• ciinsights get_connection_status reports the DB "unhealthy" (a duplicate-index error: IX_TestResults_FullyQualifiedName_CapturedAt), but every query returned data correctly — cosmetic this run, worth watching.
• Internal AzDO (def 1602): pass rate + p50/p95 computed cleanly from startTime→finishTime; aggregate-only, no job detail exposed.

Suggested prompt change (for a human to apply):

• In aspire-ci-report-common.prompt.txt → Fetch-enough-then-filter: add an explicit note that the GitHub runs REST API caps --paginate near 1000 items, so prior-window PR-validation comparisons must use a date-bounded fetch (created=START..END), not just a larger --limit/--paginate — otherwise the oldest prior-window days are silently dropped and the Δ is wrong.
• In aspire-ci-health-weekly.prompt.txt → §2 Trend: resolve the per-day basis (option (a) or (b) above). The ⚠️ basis note has now persisted across multiple runs; it should be a decision, not a standing caveat.

[radical]

🔧 Post-run review — 2026-06-20 16:25 UTC

Regressions vs previous run: none.

• Per-day trend cells Jun 03–19 are byte-identical to the prior body (verified with a row-by-row diff); only today's row (Jun 20) was computed. Jun 02 rolled off at ~18d retention as designed.
• All sections preserved (headline, lane table, trend + per-day grid, build times, reruns, infra, triage, footer). Title left untouched (--body-file only).
• The lane-table-vs-per-day basis discrepancy carried forward unchanged and is still disclosed via the ⚠️ note — not a new regression, but still unreconciled (see below).

Readability / improvement ideas:

• The basis split is the biggest wart. A reader sees PR = 80% in the lane table but ~20–46% in the per-day grid for the same week, reconciled only by a paragraph of caveat. Recommend a one-time rebaseline of the entire per-day series to direct gh-run counts, applied decisively, then frozen again — accept/reject by a human because it deliberately breaks the freeze-history rule for one run. (Flagged every run since the two bases diverged; it won't fix itself.)
• Triage rows mix two count bases. Templates/SqlClient/codegen rows use raw failCount (from topFailingTests); the EF row uses per-context broken counts. Worth a one-line "counts are raw failCount except where noted" so the magnitudes aren't read as apples-to-apples.

Data-gathering issues this run:

• ciinsights DB was unhealthy (lastError: index … 'IX_TestResults_FullyQualifiedName_CapturedAt' already exists). get_failure_summary, get_flagged_tests, search_tests all timed out initially; needed reconnect + retries. search_tests and get_failure_summary never succeeded — fell back to get_health_summary for infra + top failures.
• get_flagged_tests(problemType=broken) is incomplete as a triage source. It surfaced the EF SqlServer family but omitted the highest-volume broken tests (Templates CanNewAndBuild 189, SqlClient LoggerFactory 168, Python/Go codegen 85/84) — those appeared only in get_health_summary.topFailingTests. Trusting get_flagged_tests alone would have wrongly reported those families as "dropped out / fixed." Had to cross-check the two endpoints.
• GH Actions REST runs API caps at 1000 results per query regardless of --paginate, and a same-day range created=2026-06-15..2026-06-15 silently returned 0 rows. Reliable approach for the high-volume PR lane = fetch day-by-day with explicit …THH:MM:SSZ bounds, then dedupe by run id and bucket on created_at.
• Rerun actor split not fully separable. Only the bot's action count is solid (workflow 244898830: 91 issued / 204 skipped / 2 failed). Human-vs-bot rescue attribution needs per-rerun trigger inspection, which the list API doesn't carry — reported the aggregate (97 distinct PRs rescued) and flagged the gap rather than fabricate the split.
• Stale title. The live title is 🩺 CI Health — Weekly (Jun 12–18, 2026) — it carries a date range the prompt says should live only in the body header, and it's now two windows stale. I did not touch it (prompt: never pass --title). A human may want a one-time reset to the canonical static 🩺 CI Health — Weekly.

Suggested prompt change (for a human to apply):

1. aspire-ci-report-common.prompt.txt, data-sources/triage section: state that the broken-test triage must be built from get_health_summary.topFailingTests (raw failCount), cross-checked against get_flagged_tests(broken) — the latter alone omits high-volume families that have any passing context and will misreport them as resolved.
2. Same file, fetch section: add that the GH Actions REST runs API caps at 1000/query and that same-day created=D..D ranges can return 0 — prescribe per-day THH:MM:SSZ fetch + dedupe-by-id for the PR lane.
3. Weekly prompt, rerun section: acknowledge that human-vs-bot rescue attribution may be infeasible from the list API; permit reporting bot action-count + aggregate distinct-PR rescues with an explicit note (as done here) instead of an unverifiable split.

[radical]

🔧 Post-run review — 2026-06-20 18:30 UTC

Regressions vs previous run: none. Diffed the prior body against this one — every change is justified by genuine state movement (≈2h more data): PR validation 80%→79% (271/343), infra 336/535→341/544, PR build p50 8m→9m, rerun rescues 97→99 PRs. Frozen per-day trend cells Jun 03–19 are byte-identical; only today's (Jun 20) row was recomputed (PR 68%→62% as the partial day filled in). All six sections + header/footer preserved.

Readability / improvement ideas:

• (accept) Triage note now flags that most broken families last failed Jun 15–18 with no fresher failures — a reader should confirm "fixed vs stopped running" before triaging. This is more actionable than last run's "newly prominent" framing; keep it.
• (accept) Dropped the inline 84/79/87/81/68% direct-daily strip from the headline blockquote — it duplicated the per-day table and mixed bases inside the lead. Headline is leaner; per-day table + basis note carry shape.

Data-gathering issues this run:

• GitHub Actions runs API hard-caps at 1000 results. A single --paginate PR-validation fetch (created>=2026-06-08) truncated at exactly 1000, silently dropping Jun 08–09 (oldest returned was Jun 10). Fix: fetch high-volume lanes (PR ≈130/day) bucketed by date window (created=A..B), never one open-ended query. Verified each bucket (≤655 rows) reached its window edge.
• ciinsights DB is degraded. Connection status reported an index-creation error; get_health_summary, get_flagged_tests, and query_test_failures timed out / errored on first try. Workable path this run: get_failure_summary, scoped get_health_summary (flaky/trends off), and per-family search_tests. query_test_failures is currently erroring outright — avoid it until the DB index is repaired.
• Rerun actor split can't be derived from workflow-run conclusions. Workflow 244898830 conclusions over-count (no-op "closed PR / no retryable job" invocations conclude success/skipped) and hit the same 1000 cap. The reliable rescue metric is run_attempt>1 → success on the PR runs (99 distinct PRs rescued, ~14 net still-red), but that cannot separate 🤖 bot vs 🧑 human. This gap was also flagged last run.

Suggested prompt change (for a human to apply):

1. aspire-ci-report-common.prompt.txt → "Fetch-enough-then-filter": add an explicit warning that the Actions workflow-runs endpoint returns at most 1000 results total regardless of pagination, so PR validation must be fetched in per-window created=A..B buckets. The current "use a generous --limit" guidance is insufficient — a generous limit still silently truncates the oldest slice at 1000.

2. aspire-ci-report-common.prompt.txt → "Reruns & the flaky tax": either (a) adopt the run_attempt>1→success distinct-PR rescue count as the canonical metric and drop the bot-vs-human actor split, or (b) specify the per-run log-parse step (RETRYABLE_JOBS / closed-PR skip) needed to count real rerun API calls. Two consecutive runs have now been unable to produce the bot-vs-human split as written.

3. Trend basis — the freeze rule is now preserving known-wrong data inside the headline window. The frozen Jun 14–18 per-day cells (PR 25–41%, old differenced basis) fall within the current trailing-6d window yet contradict the §1 headline (PR 79% direct). Because new days roll in on the direct basis at the top while wrong cells age out only at ~18d, the contradiction persists for ~5 more days. Recommend a human-approved one-time full-series recompute to the direct gh-run basis (a deliberate, announced reset — not a silent per-run recompute, which the freeze rule rightly forbids), after which freeze resumes on a consistent basis. Until then the ⚠️ basis note is doing necessary load-bearing work and must stay.

   ✅ Resolved 2026-06-21: performed the one-time full-series recompute to the direct gh-run basis. The per-day table + hosted chart now match §1 (validated: 6d PR 296/369 ≈ 80% reproduces the §1 271/343 headline; 6d main 30/69 = 43% reproduces 29/67). The ⚠️ note is replaced by a ✅ basis-corrected note, and the freeze rule now pins the corrected cells going forward.