fix(coordinator): totals cache misses no longer queue behind the refresher

[brandon-eigenlabs]

Summary

A /v1/network/totals request that missed the read cache waited, unbounded, for any compute already in flight for its window, then blocked on the shared queryMu behind the refresher's other windows, then ran its own 10 s attempt. During the September outage that produced 20–40 s client latencies (multiples of the store timeout) for what was always going to be a 503, and every external poll added another provider_earnings scan while the refresher was already failing. This bounds the request-path wait on an in-flight compute to 2 s and makes the request path TryLock the query bound: if the refresher holds it, the request answers from the cache or 503s at once and never starts an aggregate. The 5-minute safety TTL and the refresher's own behavior are unchanged.

Linked issue

Closes #1155

Test plan

• cd coordinator && gofmt -l ./api — clean; go build ./... — ok; go vet ./api/ — clean
• cd coordinator && go test ./api/ -run 'Totals|Cache|Stats|Summary' -count=1 — pass
• cd coordinator && go test $(go list ./... | grep -v /internal/api) -count=1 — the pre-push set, pass
• make docs-stamp FILES=docs/reference/api-contracts.md; make docs-impact-check BASE=origin/master — coverage OK; make docs-check — 311 files OK

Tests (coordinator/api/cache_concurrency_test.go):

• TestTotalsColdRequestsDoNotQueueBehindBackgroundRefresh replaces TestTotalsColdWindowsSerializeWithBackgroundRefresh, which encoded the old contract (cold requests wait behind the refresher and all return 200). With the refresher's first window held open: a direct request-path compute returns errComputeBusy; cold requests for all four windows return 503 within coldFillWait + 1 s; the store saw exactly one call (requests started no aggregate). After the refresher finishes all windows the same requests are 200 cache hits; peak query concurrency stays 1; a failing compute still releases the bound. Fails without the fix (requests block on the held query and return 200 only after it releases).
• TestGetCachedEntryBoundsWaitOnInflightCompute — a request finding an open flight returns (nil, false) after ~coldFillWait without computing; a refresher finding the same flight keeps waiting and returns once it closes.

Components touched

• coordinator (Go)
• provider (Rust, legacy)
• provider-swift (Swift CLI)
• console-ui (Next.js)
• enclave (Swift)
• infra / CI / release
• docs

Protocol / interface changes

• No protocol/interface changes
• Yes — described above and matching side updated

No new status codes or shapes. The observable change is latency: a miss during an in-flight refresh now resolves within 2 s (cache body or the existing 503 envelope) instead of after the refresher's and its own store timeouts. docs/reference/api-contracts.md states this on the /v1/network/totals row and in the cache-behavior paragraph. /v1/stats shares computeCachedEntry, so its miss path gets the same bounded wait (it has no separate query lock; unchanged otherwise).

Documentation impact

• Canonical documentation updated
• No documentation needed — reason:

docs/reference/api-contracts.md (/v1/network/totals row; cache-behavior paragraph names coldFillWait), stamp refreshed; CHANGELOG.md bullet under Unreleased.

Notes for reviewers

Before / after — observable behavior

sequenceDiagram
    participant C as client (cache miss)
    participant H as handleNetworkTotals
    participant F as in-flight refresh (window A)
    participant Q as queryMu (held by refresher)
    participant DB as provider_earnings
    Note over C,DB: BEFORE
    C->>H: GET ?window=B
    H->>Q: Lock() … waits for refresher's window (≤10 s)
    Q-->>H: acquired
    H->>DB: own aggregate (≤10 s)
    DB--xH: timeout
    H-->>C: 503 after 20–40 s; one extra scan added

Loading

sequenceDiagram
    participant C as client (cache miss)
    participant H as handleNetworkTotals
    participant F as in-flight refresh
    participant Q as queryMu (held by refresher)
    Note over C,Q: AFTER
    C->>H: GET ?window=A (flight open)
    H->>F: wait ≤ coldFillWait (2 s)
    H-->>C: cached body, or 503, within 2 s
    C->>H: GET ?window=B (no flight)
    H->>Q: TryLock()
    Q-->>H: busy → errComputeBusy (not logged, not counted)
    H-->>C: cached body, or 503, immediately

Loading

Code flow

flowchart TD
    R[request miss] --> G["getCachedEntry (refresh=false)"]
    G -->|flight open| W["select: flight done / coldFillWait"] --> C1[readCache.Get → body or 503]
    G -->|no flight| K["computeNetworkTotals(window, waitForQuery=false)"]
    K -->|"TryLock ok"| DB[aggregate → Set → 200]
    K -->|"TryLock busy"| B[errComputeBusy] --> C1
    BG[refresher] --> G2["refreshCachedEntry (refresh=true)"] -->|flight open| W2["<-wait (unbounded, as before)"]
    G2 -->|no flight| K2["computeNetworkTotals(window, true) → Lock()"]

Loading

Design choices

• Bound, don't skip, the in-flight wait. On a healthy system a compute finishes in well under 2 s, so a request arriving mid-flight still gets the coalesced fresh body; only a compute heading for the 10 s store timeout is abandoned. Skipping the wait entirely would have turned every cold-start race into a 503.
• errComputeBusy is not a refresh failure. computeCachedEntry returns the cached value for it without the cache refresh failed log or the cache.refresh_failed counter, so request traffic cannot inflate the monitor observability: Datadog monitor on cache.refresh_failed as code #1157 adds.
• TTL left alone. network totals: inbound cache misses queue behind the background refresher on queryMu, turning a 10 s timeout into 20–40 s client latency #1155 also raises whether the 5-minute "keep last good" TTL should give way to serving stale with updated_at. That is a product decision about how old a number the dashboard may show as current; this PR only removes the queueing and the request-driven load, and keeps the documented contract.
• coldFillWait is a package var only so the tests can shorten it; it is not configurable.

Limitations

• Not exercised against production; the behavior is pinned by the httptest-level test with a blocking store. After deploy, an outage should show /v1/network/totals misses resolving in ≤2 s (curl -w %{time_total}) instead of 20–40 s, and no request-driven NetworkTotals queries in pg_stat_activity while the refresher is running.
• No milestone set (open milestones are v0.3.6/v0.4.0 with zero items).

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[vercel]

@brandon-eigenlabs is attempting to deploy a commit to the EigenLabs Team on Vercel.

A member of the Team first needs to authorize it.