content-review: close the loop between the claims detector and the fixer (#20968)

* content-review: close the loop between the claims detector and the fixer

The nightly claims re-verification finds real bugs and the per-article
review worker never acts on them. Observed end-to-end in #20927:
concepts/providers/_index.md was flagged 2026-08-15 for a contradicted
version pin, boosted to the front of the queue on that marker, reviewed,
reported "0 contradicted" across 74 re-extracted claims, and merged a
one-line unrelated repair — with the flagged bug still on master (it took
a hand-written PR, #20961). The marker then cleared, so the entity
returned to the pool to be re-flagged and re-missed on a ~2-day cycle.

Three defects, three fixes:

1. The evidence never reached the worker. select-articles.py put only
   `len(stale_claims)` — an integer — into the queue item, while the
   review skill was told to "treat the ledger markers' entity keys and
   evidence as priority findings". It was being asked to act on data it
   was never given, so it re-derived from scratch and missed. The queue
   item now carries `stale_claim_markers` in full, and the skill
   documents them as must-address findings.

2. Unresolved markers were silently cleared. build_record rebuilt the
   ledger entry from the queue and dropped `stale_claims` whether or not
   anything had been done. A marker now retires only when the verdict
   names its entity in the new optional `resolved_claims` field; anything
   else carries forward with `unresolved_reviews` incremented. Since an
   unboundedly-boosted page would starve the queue, a marker that
   survives MARKER_ESCALATION_CAP (2) reviews is flagged `escalated`:
   still recorded and still visible, but no longer boosting — the signal
   that a human, not another identical pass, is the next step.

3. Markers on generated pages could never be cleared at all.
   select-articles.py globs content/docs/**/*.md, so an adapter-generated
   page (the pre-built policy-pack tables) can never be selected — while
   already_marked() excluded its entities from re-verification forever.
   Three entities were lost this way over five nights
   (iam-user-unused-credentials-90, redshift-backup-enabled,
   cloud-logging-retention-period-365). apply_markers() now skips pages
   with no source file and reports the verdict instead, leaving the
   entity in the pool; those contradictions are real but the fix is
   upstream of this repo.

make test-review-pipeline and make lint pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hp4LqiP4u81uT48MS8Hea4

* content-review: keep escalated markers on the ledger through a review

Review feedback on #20968 (thanks @unblocked) found a real hole in the
escalation path, and a smaller inconsistency next to it.

Escalated markers were filtered out of the queue item, but
record-review.py rebuilds the ledger entry *from the queue* and writes
back only what survives — so the first time an escalated marker's page
was reviewed for any other reason (ordinary staleness, or --paths), that
marker was dropped from the ledger entirely. already_marked() in
reverify-claims.py would then stop seeing the entity and let it back into
the nightly pool, restarting the detect/boost/miss/clear cycle this PR
exists to break, just two reviews later. The PR body claimed the opposite
invariant ("the marker stays on the ledger entry"), and it did not hold.

Relatedly, the queue's `stale_claims` count was computed over all markers
while `stale_claim_markers` carried only the non-escalated ones, so a page
whose only marker was escalated advertised `stale_claims: 1` alongside an
empty list — contradicting what SKILL.md tells the worker those two fields
mean.

Both come from the same root cause: escalation was being enforced at the
wrong layer. It is a *boost* policy, not a visibility policy. So the queue
item now carries every marker (new all_markers()) and the count is taken
over that same list, while active_markers() — non-escalated only — is used
solely for the scoring boost. Escalated markers travel, persist, and stay
resolvable; they simply stop jumping the page to the front of the queue.
SKILL.md documents the `escalated` flag so the worker knows what it means
rather than inferring it.

Regression coverage for the exact failure: an escalated marker rides the
queue item, reaches a --paths review, survives build_record with its
counter still incrementing, and can still be retired via resolved_claims;
plus the count/list agreement. All three new select-articles checks fail
against the previous code and pass against this one.

make test-review-pipeline and make lint pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hp4LqiP4u81uT48MS8Hea4

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: CamSoper

.claude/commands/review-existing-content/SKILL.md

@@ -52,6 +52,15 @@ Read `.content-review-queue.json` from the repo root (written by
         "feedback": { "yes": 4, "no": 9, "neg_rate": 0.6923, "multiplier": 1.27 }
       },
       "last_reviewed": null,
+      "stale_claims": 1,
+      "stale_claim_markers": [
+        { "entity_key": "version/pulumi-package",
+          "verdict": "contradicted",
+          "evidence": "CHANGELOG lists \"Save package source to `packages` in Pulumi.yaml on `package add`\" under 3.163.0, not 3.157.0.",
+          "source": "gh api repos/pulumi/pulumi/releases",
+          "checked_at": "2026-08-15",
+          "unresolved_reviews": 0 }
+      ],
       "score": 0.91 }
   ]
 }
@@ -60,8 +69,31 @@ Read `.content-review-queue.json` from the repo root (written by
 - `lane` — `priority` (scored pick) or `manual` (workflow_dispatch override).
 - `stale_claims` (when present) — count of this page's volatile claims the
   nightly re-verification found contradicted (see §Claims index below). A
-  non-zero count is why the page jumped the queue: treat the ledger markers'
-  entity keys and evidence as priority findings to re-check first.
+  non-zero count is why the page jumped the queue.
+- `stale_claim_markers` (when present) — **those findings in full**, each with
+  the `entity_key`, the `verdict`, the `evidence` the nightly verifier
+  recorded, the `source` it reached, and `unresolved_reviews` (how many prior
+  reviews saw this marker and left it unresolved). **These are the highest-
+  priority findings in your queue and you must address every one of them.**
+  The nightly lane has already done the expensive part — it identified the
+  entity, reached an authoritative source, and wrote down what that source
+  says — so start here rather than waiting to see whether your own claim
+  extraction happens to re-derive the same finding. It may not: a page boosted
+  for a contradicted version pin was once reviewed, reported "0 contradicted"
+  across 74 re-extracted claims, and merged an unrelated one-line repair while
+  the flagged bug stayed on master.
+  A marker carrying `"escalated": true` has already survived two reviews
+  unresolved, so it no longer jumps the page to the front of the queue and a
+  human may be looking at it — but it is still a live finding and still yours
+  to resolve if you can. Treat it like any other marker.
+  For each marker, either apply the fix, or establish that the flag was wrong
+  (the nightly verdicts are single-sample and do produce false positives —
+  a synthetic module path for a locally generated SDK was once flagged as a
+  broken import). Then list its `entity_key` in the verdict sentinel's
+  `resolved_claims`. A marker you do not resolve is carried onto the next
+  review with `unresolved_reviews` incremented, and after two such rounds it
+  is escalated for a human — so silently skipping one does not make it go
+  away, it just delays it.
 - `no_retire` — when true, retirement must never be proposed for this page.
   This is the **hard veto** on retirement — honor it regardless of evidence.
 - `reader_signals` / `signals` — Search Console and feedback-widget figures
@@ -335,6 +367,7 @@ the canonical ledger record, and uploads it to S3 keyed by slug.
   "skipped_findings": 2,
   "retirement": false,
   "clarity_flag": true,
+  "resolved_claims": ["version/pulumi-package"],
   "applied": [
     { "category": "claim", "file": "content/docs/iac/concepts/stacks/_index.md",
       "lines": [42, 43], "source": "verified-claims:c3" },
@@ -362,6 +395,12 @@ the canonical ledger record, and uploads it to S3 keyed by slug.
   `len(applied)`. The workflow's scope gate cross-checks these against the
   artifacts and the branch diff; for link fixes (which have no artifact) the
   declared lines must actually carry the link in the pre-fix file.
+- `resolved_claims`: optional; the `entity_key` of every
+  `stale_claim_markers` entry you resolved this run — fixed, or shown to be a
+  false positive (say which, in the PR body). Omit or leave empty when the
+  queue item carried no markers. Anything you leave out is carried forward to
+  the next review rather than cleared, so this list is the only way a marker
+  retires.
 - `clarity_flag`: optional; `true` when you flagged a readthrough `reconception`
   for this page. Carries onto the ledger record so the page's structural
   follow-up is durable even when the verdict is `clean` or `fixed` (the

scripts/content-review/record-review.py

@@ -130,6 +130,9 @@ def load_queue_article(queue_path: Path) -> dict:
         # verbatim from the queue entry; null on a signal-blind run.
         "signals": a.get("signals"),
         "signals_available": bool((data.get("reader_signals") or {}).get("available")),
+        # Stale-claim markers in full (entity_key/verdict/evidence/source/...).
+        # carry_markers() needs them to decide which survive this review.
+        "stale_claim_markers": a.get("stale_claim_markers") or [],
     }
 
 
@@ -243,6 +246,43 @@ def canonical_branch_pushed(slug: str) -> bool:
 # ---- derivation -------------------------------------------------------------
 
 
+# A stale-claim marker survives this many reviews that saw it and did not
+# resolve it; after that it is kept but flagged `escalated`, which stops
+# select-articles.py boosting the page (see MARKER_ESCALATION_CAP there).
+MARKER_ESCALATION_CAP = 2
+
+
+def carry_markers(article: dict, verdict: dict | None) -> list[dict]:
+    """Stale-claim markers that outlive this review, with their retry state.
+
+    The nightly re-verification writes a marker when a volatile claim goes
+    contradicted; the marker boosts the page into this worker's queue. Before
+    this function existed the marker simply vanished when the review's ledger
+    record landed, whether or not anything had been done about it — so a review
+    that missed the finding silently cleared the flag and the entity went back
+    into the pool to be re-flagged, re-boosted, and re-missed on a ~2-day cycle
+    (observed end-to-end in pulumi/docs#20927).
+
+    A marker is retired only when the verdict names its entity in
+    `resolved_claims` — the worker asserting it either fixed the claim or
+    determined the flag was wrong. Everything else is carried forward with
+    `unresolved_reviews` incremented, and flagged `escalated` once that count
+    reaches MARKER_ESCALATION_CAP so a human is the next step rather than
+    another identical pass.
+    """
+    resolved = {str(k) for k in ((verdict or {}).get("resolved_claims") or [])}
+    carried: list[dict] = []
+    for m in article.get("stale_claim_markers") or []:
+        if not isinstance(m, dict):
+            continue
+        if m.get("entity_key") in resolved:
+            continue
+        seen = int(m.get("unresolved_reviews") or 0) + 1
+        carried.append({**m, "unresolved_reviews": seen,
+                        "escalated": seen >= MARKER_ESCALATION_CAP})
+    return carried
+
+
 def build_record(article: dict, verdict: dict | None, pr: dict | None,
                  slug: str, claude_succeeded: bool = False,
                  branch_exists: bool = False) -> dict:
@@ -285,6 +325,9 @@ def build_record(article: dict, verdict: dict | None, pr: dict | None,
         "signals_available": bool(article.get("signals_available")),
         "reviewed_at": datetime.now(timezone.utc).date().isoformat(),
     }
+    markers = carry_markers(article, verdict)
+    if markers:
+        rec["stale_claims"] = markers
 
     if verdict is None:
         if claude_succeeded and not branch_exists:
@@ -456,6 +499,64 @@ def check(name, cond):
         check("signal-blind queue -> signals null, signals_available False",
               blind_article["signals"] is None
               and blind_article["signals_available"] is False)
+        # --- stale-claim marker retention -------------------------------
+        marker = {"entity_key": "version/pulumi-package", "verdict": "contradicted",
+                  "evidence": "CHANGELOG says 3.163.0", "source": "gh release view",
+                  "checked_at": "2026-08-15"}
+        marked = {**article, "stale_claim_markers": [marker]}
+
+        rec_m = build_record(marked, {"verdict": "clean", "reason": "accurate"},
+                             None, marked["slug"])
+        check("a review that resolves nothing carries the marker forward",
+              [m["entity_key"] for m in rec_m["stale_claims"]] == ["version/pulumi-package"])
+        check("carried marker counts the miss",
+              rec_m["stale_claims"][0]["unresolved_reviews"] == 1
+              and rec_m["stale_claims"][0]["escalated"] is False)
+        check("carried marker keeps its evidence",
+              rec_m["stale_claims"][0]["evidence"] == "CHANGELOG says 3.163.0")
+
+        rec_r = build_record(marked, {"verdict": "fixed", "fixes": 1,
+                                      "resolved_claims": ["version/pulumi-package"]},
+                             None, marked["slug"])
+        check("a resolved marker retires", "stale_claims" not in rec_r)
+
+        second = {**article, "stale_claim_markers": [rec_m["stale_claims"][0]]}
+        rec_e = build_record(second, {"verdict": "clean", "reason": "x"},
+                             None, second["slug"])
+        check("second unresolved review escalates the marker",
+              rec_e["stale_claims"][0]["unresolved_reviews"] == 2
+              and rec_e["stale_claims"][0]["escalated"] is True)
+
+        # The other half of the round-trip: select-articles.py hands escalated
+        # markers through, so they must survive rather than be filtered out
+        # here — otherwise the ledger loses them and the entity re-enters the
+        # nightly pool.
+        esc_marker = {**marker, "entity_key": "version/old-miss",
+                      "unresolved_reviews": 2, "escalated": True}
+        with_esc = {**article, "stale_claim_markers": [marker, esc_marker]}
+        rec_esc = build_record(with_esc, {"verdict": "clean", "reason": "x"},
+                               None, with_esc["slug"])
+        carried = {m["entity_key"]: m for m in rec_esc["stale_claims"]}
+        check("escalated marker persists through a review",
+              "version/old-miss" in carried)
+        check("escalated marker stays escalated and keeps counting",
+              carried["version/old-miss"]["escalated"] is True
+              and carried["version/old-miss"]["unresolved_reviews"] == 3)
+        check("an escalated marker can still be resolved",
+              "stale_claims" not in build_record(
+                  {**article, "stale_claim_markers": [esc_marker]},
+                  {"verdict": "fixed", "fixes": 1,
+                   "resolved_claims": ["version/old-miss"]},
+                  None, article["slug"]))
+
+        rec_none = build_record(article, {"verdict": "clean", "reason": "x"},
+                                None, article["slug"])
+        check("no markers -> no stale_claims key", "stale_claims" not in rec_none)
+
+        rec_incomplete = build_record(marked, None, None, marked["slug"])
+        check("marker survives an incomplete run too",
+              rec_incomplete["stale_claims"][0]["unresolved_reviews"] == 1)
+
         blind_rec = build_record(blind_article, None, None, blind_article["slug"])
         check("signal-blind record persists null/false",
               blind_rec["signals"] is None and blind_rec["signals_available"] is False)

scripts/content-review/reverify-claims.py

@@ -233,7 +233,8 @@ def load_ledger(ledger_dir: Path) -> dict[str, dict]:
     return entries
 
 
-def apply_markers(ledger: dict[str, dict], stale: list[dict], today: date) -> dict[str, dict]:
+def apply_markers(ledger: dict[str, dict], stale: list[dict], today: date,
+                  repo_root: Path | None = None) -> dict[str, dict]:
     """Fold stale entity verdicts into the affected pages' ledger entries.
 
     Returns {slug: updated entry (without bookkeeping keys)} for every entry
@@ -243,7 +244,19 @@ def apply_markers(ledger: dict[str, dict], stale: list[dict], today: date) -> di
     not synthesized: every changed entry is uploaded to the same S3 key
     record-review.py owns, so a two-field stub would silently destroy that
     page's status / reviewed_at / attempts / tier / score. The entity simply
-    stays unmarked and comes back around on a later rotation."""
+    stays unmarked and comes back around on a later rotation.
+
+    A page with no markdown source on disk is skipped for a different and more
+    important reason. `select-articles.py` builds its candidate set by globbing
+    `content/docs/**/*.md`, so a page rendered by a Hugo content adapter (the
+    pre-built policy-pack tables, generated from `data/`) can never be selected
+    for review — which means a marker written there can never be cleared, and
+    `already_marked()` would then exclude that entity from re-verification
+    permanently. Three entities were lost that way over five nights before this
+    guard existed. Such a contradiction is real and worth acting on, but the
+    action is upstream of this repo (fix the `data/` source or the product
+    metadata behind it), so the verdict is reported and left in the pool rather
+    than converted into a marker nothing can retire."""
     changed: dict[str, dict] = {}
     for s in stale:
         marker = {
@@ -254,6 +267,12 @@ def apply_markers(ledger: dict[str, dict], stale: list[dict], today: date) -> di
             "checked_at": today.isoformat(),
         }
         for page in s["pages"]:
+            if repo_root is not None and not (repo_root / page["path"]).is_file():
+                warn(
+                    f"{page['path']} has no source file (generated page); reporting "
+                    f"{s['entity_key']} without a marker no review could ever clear"
+                )
+                continue
             entry = ledger.get(page["path"])
             if entry is None:
                 warn(
@@ -395,7 +414,7 @@ def check_entity(key: str) -> dict:
     report["meta"]["n_demoted"] = sum(1 for r in results if r["demoted_from"])
 
     if stale:
-        changed = apply_markers(ledger, stale, today)
+        changed = apply_markers(ledger, stale, today, repo_root)
         uri = os.environ.get("CONTENT_REVIEW_LEDGER_URI", "").strip()
         for slug, entry in sorted(changed.items()):
             local = Path(args.ledger_dir) / f"{slug}.json"
@@ -413,6 +432,7 @@ def check_entity(key: str) -> dict:
 
 
 def self_test() -> int:
+    import tempfile
     failures = []
 
     def check(name, cond):
@@ -514,6 +534,31 @@ def snap(slug, reviewed_at, *claims):
     check("re-marking is idempotent",
           len(changed2["docs-a"]["stale_claims"]) == 1)
 
+    # Generated pages have no markdown source, so select-articles.py can never
+    # queue them and a marker there could never be cleared. Report, don't mark.
+    with tempfile.TemporaryDirectory() as td:
+        root = Path(td)
+        (root / "content/docs").mkdir(parents=True)
+        (root / "content/docs/a.md").write_text("real page\n")
+        gen = [{"entity_key": "numerical/pack-count", "verdict": "contradicted",
+                "evidence": "", "source": "",
+                "pages": [{"path": "content/docs/reference/generated.md",
+                           "slug": "docs-reference-generated"}]}]
+        led = {"content/docs/reference/generated.md": {
+            "path": "content/docs/reference/generated.md",
+            "slug": "docs-reference-generated", "status": "clean"}}
+        check("sourceless page gets no marker",
+              apply_markers(led, gen, today, root) == {})
+        check("sourceless page's ledger entry is left untouched",
+              "stale_claims" not in led["content/docs/reference/generated.md"])
+        # ...while a page that does exist on disk is still marked normally.
+        led2 = {"content/docs/a.md": {"path": "content/docs/a.md",
+                                      "slug": "docs-a", "status": "clean"}}
+        on_disk = [{**gen[0], "pages": [{"path": "content/docs/a.md",
+                                         "slug": "docs-a"}]}]
+        check("page with a source file is still marked",
+              set(apply_markers(led2, on_disk, today, root)) == {"docs-a"})
+
     check("already_marked sees the marker",
           already_marked("version/pulumi-gcp", ents["version/pulumi-gcp"], ledger))
     check("already_marked ignores other entities",
@@ -522,7 +567,6 @@ def snap(slug, reviewed_at, *claims):
 
     # Health-observation meta: the early-exit paths must say why they stopped
     # (signal-health.py's reverify signal reads these fields).
-    import tempfile
 
     def run_report(d: Path, extra_env_unset: list[str]) -> dict:
         saved = {k: os.environ.pop(k) for k in extra_env_unset if k in os.environ}