What happens
store.Leaderboard returns a bare []LeaderboardRow and swallows every query error (coordinator/store/postgres.go, if err != nil { return nil }). When the three-CTE aggregate over provider_earnings exceeds its 10 s context timeout, the handler receives an empty slice, serializes it as 200 {"entries":[]} and caches that body for five minutes as a successful board.
Observed live on production on 2026-09-18 against api.darkbloom.dev:
GET /v1/leaderboard?metric=earnings&window=24h -> 200 0.04s 50 entries (cache hit)
GET /v1/leaderboard?metric=earnings&window=all -> 200 10.04s 0 entries (cache miss -> timeout)
GET /v1/leaderboard?metric=jobs&window=24h -> 200 10.04s 0 entries (cache miss -> timeout)
GET /v1/network/totals?window=all -> 503 17.02s (same table, same instant)
The 10.04 s latency is the store timeout hit to the millisecond. An empty leaderboard is therefore currently ambiguous between "no provider earned anything" and "the query timed out", and any downstream consumer will plot the latter as zero.
Why it matters
/v1/network/totals hitting the same timeout on the same table answers 503 service_unavailable (handleNetworkTotals: "say so rather than serve a zero row as if it were data"). The two public endpoints disagree about the same outage.- The failure is silent: the 503 got investigated; the empty board was served as a measurement for days.
- Scan errors are also swallowed (
continue), so a partially-read board can be published as complete.
Expected
store.Leaderboard returns ([]LeaderboardRow, error) on both backends; handleLeaderboard answers 503 service_unavailable on error and does not cache it, matching handleNetworkTotals and handleNetworkSeries. A genuinely empty window is still a successful, cacheable 200 with empty entries.
Not in scope here
The underlying cost of the aggregate (~140M-row provider_earnings with no created_at-leading index, so 24h/7d windows are parallel seq scans and all is a full index scan) is a separate performance issue. This issue is only about reporting the failure honestly.
What happens
store.Leaderboardreturns a bare[]LeaderboardRowand swallows every query error (coordinator/store/postgres.go,if err != nil { return nil }). When the three-CTE aggregate overprovider_earningsexceeds its 10 s context timeout, the handler receives an empty slice, serializes it as200 {"entries":[]}and caches that body for five minutes as a successful board.Observed live on production on 2026-09-18 against
api.darkbloom.dev:The 10.04 s latency is the store timeout hit to the millisecond. An empty leaderboard is therefore currently ambiguous between "no provider earned anything" and "the query timed out", and any downstream consumer will plot the latter as zero.
Why it matters
/v1/network/totalshitting the same timeout on the same table answers 503service_unavailable(handleNetworkTotals: "say so rather than serve a zero row as if it were data"). The two public endpoints disagree about the same outage.continue), so a partially-read board can be published as complete.Expected
store.Leaderboardreturns([]LeaderboardRow, error)on both backends;handleLeaderboardanswers 503service_unavailableon error and does not cache it, matchinghandleNetworkTotalsandhandleNetworkSeries. A genuinely empty window is still a successful, cacheable 200 with empty entries.Not in scope here
The underlying cost of the aggregate (~140M-row
provider_earningswith nocreated_at-leading index, so 24h/7d windows are parallel seq scans andallis a full index scan) is a separate performance issue. This issue is only about reporting the failure honestly.