docs-review: prefer English documentation sources in the verifier (#20894)

Claims-reverify run #36's one contradicted verdict cited the Traditional
Chinese localization of the AWS docs (docs.aws.amazon.com/zh_tw/...) and
quoted its evidence in Chinese inside an otherwise-English report. The
pass3 verify lane does no client-side fetching — it uses Anthropic's
server-side web_search tool — so the search engine's locale choice
propagated straight into the evidence.

Three layers of fix:

* WEB_SEARCH_TOOL now sets user_location {approximate, US}, biasing the
  server-side search away from localized doc variants.
* The pass3 source-order prompt and route header instruct the verifier
  to treat English doc pages as canonical: strip locale segments
  (/zh_tw/ dropped, /ja-jp/ -> /en-us/, ?hl= dropped) from cited URLs
  and quote evidence in English.
* The pass2 lane's real fetch layer (extract-urls-and-fetch.py) now
  normalizes known-localized URLs on docs.aws.amazon.com,
  learn.microsoft.com, and cloud.google.com to their English canonical
  form before fetching, and sends Accept-Language: en-US on every fetch
  for hosts that content-negotiate.

make test-review-pipeline passes.


Claude-Session: https://claude.ai/code/session_01Hp4LqiP4u81uT48MS8Hea4

Co-authored-by: Claude <noreply@anthropic.com>

Author: CamSoper

.claude/commands/docs-review/scripts/extract-urls-and-fetch.py

@@ -57,6 +57,17 @@
     hub.docker.com/r/<o>/<r>              → hub.docker.com/v2/repositories/<o>/<r>/tags/?page_size=20
     hub.docker.com/_/<r>                  → hub.docker.com/v2/repositories/library/<r>/tags/?page_size=20
 
+Locale normalization: the big cloud-provider doc sites serve localized
+variants that authors (and search engines) sometimes link, which makes the
+verifier quote non-English evidence against English claims. Known-localized
+URLs are rewritten to their English canonical form before fetching, and
+every fetch sends `Accept-Language: en-US,en` for hosts that content-
+negotiate:
+
+    docs.aws.amazon.com/<ll_cc>/<path>    → docs.aws.amazon.com/<path>
+    learn.microsoft.com/<ll-cc>/<path>    → learn.microsoft.com/en-us/<path>
+    cloud.google.com/<path>?hl=<lang>     → cloud.google.com/<path>  (hl= dropped)
+
 The `url` field in the record stays the original (so verifier pattern-
 matching against the diff text still works); a `fetched_url` field appears
 only when normalization changed the target.
@@ -72,6 +83,7 @@
 import sys
 import time
 import urllib.error
+import urllib.parse
 import urllib.request
 from pathlib import Path
 
@@ -102,9 +114,30 @@
     r"^https://hub\.docker\.com/_/([^/\s#?]+)(?:/[^\s#?]*)?"
 )
 
+# Localized doc variants → English canonical (see module docstring). AWS and
+# Microsoft encode the locale as the first path segment (`zh_tw`, `ja-jp`);
+# Google Cloud uses an `?hl=` query parameter. Anchored to the exact hosts so
+# an unrelated two-letter path segment elsewhere is never touched.
+AWS_LOCALE_RE = re.compile(r"^(https://docs\.aws\.amazon\.com)/[a-z]{2}_[a-z]{2}/(.+)$")
+MSFT_LOCALE_RE = re.compile(
+    r"^(https://learn\.microsoft\.com)/(?!en-us(?:/|$))[a-z]{2}-[a-z]{2}/(.+)$",
+    re.IGNORECASE,
+)
+GCLOUD_HOST = "https://cloud.google.com/"
+
+
+def _strip_gcloud_hl(url: str) -> str:
+    parts = urllib.parse.urlsplit(url)
+    query = [
+        (k, v)
+        for k, v in urllib.parse.parse_qsl(parts.query, keep_blank_values=True)
+        if k != "hl"
+    ]
+    return urllib.parse.urlunsplit(parts._replace(query=urllib.parse.urlencode(query)))
+
 
 def normalize_url(url: str) -> str:
-    """Rewrite known SPA / JS-rendered URLs to content-API equivalents.
+    """Rewrite known SPA / JS-rendered and localized URLs to canonical form.
 
     Returns the URL unchanged if no rewrite applies. See module docstring
     for the full pattern list.
@@ -121,6 +154,14 @@ def normalize_url(url: str) -> str:
     if m:
         (repo,) = m.groups()
         return f"https://hub.docker.com/v2/repositories/library/{repo}/tags/?page_size=20"
+    m = AWS_LOCALE_RE.match(url)
+    if m:
+        return f"{m.group(1)}/{m.group(2)}"
+    m = MSFT_LOCALE_RE.match(url)
+    if m:
+        return f"{m.group(1)}/en-us/{m.group(2)}"
+    if url.startswith(GCLOUD_HOST) and "hl=" in url:
+        return _strip_gcloud_hl(url)
     return url
 
 
@@ -218,7 +259,13 @@ def fetch_one(url: str) -> dict:
     if fetched != url:
         record["fetched_url"] = fetched
     try:
-        req = urllib.request.Request(fetched, headers={"User-Agent": USER_AGENT})
+        req = urllib.request.Request(
+            fetched,
+            # Accept-Language keeps content-negotiating hosts from serving a
+            # localized variant the locale normalization above didn't know
+            # about.
+            headers={"User-Agent": USER_AGENT, "Accept-Language": "en-US,en;q=0.9"},
+        )
         with urllib.request.urlopen(req, timeout=FETCH_TIMEOUT) as resp:
             status = getattr(resp, "status", 200)
             raw = resp.read(CONTENT_TEXT_CAP * 4)  # over-read; HTML-strip below

.claude/commands/docs-review/scripts/verify-claims.py

@@ -258,8 +258,18 @@
 }
 
 # Anthropic server-side web search; the API runs the search and returns results
-# inline, so no client round-trip is needed for the search itself.
-WEB_SEARCH_TOOL = {"type": "web_search_20250305", "name": "web_search", "max_uses": 3}
+# inline, so no client round-trip is needed for the search itself. user_location
+# anchors the search to a US/English context — without it the engine can serve
+# localized doc variants (claims-reverify run #36's one contradicted verdict
+# cited docs.aws.amazon.com/zh_tw/... and quoted its evidence in Traditional
+# Chinese, which makes the report hard to audit). Belt and suspenders with the
+# English-sources instruction in the pass3 prompt below.
+WEB_SEARCH_TOOL = {
+    "type": "web_search_20250305",
+    "name": "web_search",
+    "max_uses": 3,
+    "user_location": {"type": "approximate", "country": "US"},
+}
 
 ALLOWED_GH_SUBCOMMANDS = {"search", "api", "release", "issue", "pr"}
 _SHELL_META_RE = re.compile(r"[|;&`$\\]|\$\(")
@@ -299,7 +309,7 @@
    - **Linked implementing change** — when the claim is about a NEW pulumi symbol you can't find on the default branch AND this PR cites an implementing change (a "This docs PR cites implementing change(s)" line in the user message, or a `pulumi/<repo>#<n>` / `github.com/pulumi/<repo>/(pull|commit)/...` reference), read it: `gh pr diff <n> -R pulumi/<repo>` or `gh api repos/pulumi/<repo>/commits/<sha>`. Confirmed there, the symbol is `verified`/`medium` ("not yet on default branch / released") — NOT `unverifiable`; "not in the published reference yet" is a lag, not a doubt. Docs shipping alongside a feature are the normal case.
    `gh` results count as `high` confidence when they directly match — they read source-of-truth. Don't loop `issues`/`pulls` for *blind* context discovery (a PR THIS docs PR cites is not blind — see above). Keep your `gh_query` + `read_file` calls under 8 total; if you can't close the claim, return `unverifiable` (or, from a pass1 lane, set `route_escalation: "pass3"` when a public web source plausibly could resolve it).
 3. **Pre-fetched URL** (pass2 lane) — the cited URL's content (HTTP status + body) is in the user message. Do NOT try to fetch it again. Read the body, find the supporting passage, run the framing check. If the status is not 2xx (dead link / soft-404) → `contradicted` with `evidence: "cited URL returns HTTP <status>"` and `source: "<url>"`; do NOT return `unverifiable` for a dead Pass-2 URL — a broken citation is a contradiction the author must fix. If the body is 2xx but doesn't contain the supporting passage → `unverifiable` (note the page was fetched but didn't address the claim).
-4. **Web search** (pass3 lane) — use the `web_search` tool with a query derived from the claim, then read the results. For numerical claims (prices, rates, limits), cross-check the YEAR of any page you rely on — a stale cached price is a `contradicted` when the current figure differs. If no result addresses the claim, return `unverifiable` and set `source` to `WebSearch ran query "<your query>"; top results didn't address the claim`. Reserve `unverifiable` for genuinely unfetchable claims, not "I didn't try".
+4. **Web search** (pass3 lane) — use the `web_search` tool with a query derived from the claim, then read the results. Use English-language sources: major doc sites serve localized variants (`docs.aws.amazon.com/zh_tw/...`, `learn.microsoft.com/ja-jp/...`, `cloud.google.com/...?hl=de`), and evidence quoted from one is hard to audit in an English report. When a result lands on a localized page, treat the English page as canonical — cite the URL with the locale segment removed (`/zh_tw/` dropped, `/ja-jp/` → `/en-us/`, `?hl=` dropped) and quote the evidence passage in English. For numerical claims (prices, rates, limits), cross-check the YEAR of any page you rely on — a stale cached price is a `contradicted` when the current figure differs. If no result addresses the claim, return `unverifiable` and set `source` to `WebSearch ran query "<your query>"; top results didn't address the claim`. Reserve `unverifiable` for genuinely unfetchable claims, not "I didn't try".
 
 # Cited-claim framing check (pass2 and pass3, any claim that cited a source)
 
@@ -336,7 +346,9 @@
     "pass2": ("ROUTE: pass2 (external; cited URL pre-fetched). Tools: verify_claim only — the URL's content is in the user "
               "message; do NOT re-fetch. Run the framing check and emit verify_claim. Dead/non-2xx URL → `contradicted`."),
     "pass3": ("ROUTE: pass3 (external; no pre-fetched URL). Tools: web_search, verify_claim. Search, read the results, "
-              "cross-check the YEAR on numerical claims, then emit verify_claim. If the claim turns out to describe "
+              "cross-check the YEAR on numerical claims, then emit verify_claim. Cite English-language doc pages — "
+              "strip locale segments (`/zh_tw/`, `/ja-jp/`, `?hl=`) from cited URLs and quote evidence in English. "
+              "If the claim turns out to describe "
               "Pulumi's own product/CLI behavior (default limits, rotation policies, flag semantics — even when no "
               "pulumi-shaped token appears in the text), web search cannot read product source: emit verify_claim with "
               "`route_escalation: \"pass1\"` instead of `unverifiable`."),