Skip to content

perf: serve all-time /v1/network/totals and /v1/leaderboard from earnings_summary instead of scanning provider_earnings #1143

Description

@brandon-eigenlabs

What happens

window=all is the default for GET /v1/network/totals (empty or lifetime map to it) and the most expensive window: NetworkTotals and Leaderboard run three CTEs over provider_earnings (~140M rows / 33 GB on production as of 2026-09-19). On the read replica the all-time totals plan is a full Index Scan over idx_provider_earnings_account (~144M entries, planner cost ~29M) and exceeds the 10 s store timeout, which is why /v1/network/totals?window=all has answered 503 since 2026-09-10 (and the leaderboard cache misses timed out, #1139). #1141 addresses the short windows with an index; no index helps a full-table aggregate.

What already exists

earnings_summary holds lifetime total_count, total_micro_usd, total_prompt_tokens, total_completion_tokens per account and per provider, maintained in the same transaction as every earnings write (RecordProviderEarning, creditProviderAccount, SettleProviderFloorDraw) and backfilled once by migrateEarningsSummary. On the replica the account and provider partitions agree exactly and track provider_earnings within the ingestion delta. Reading it costs ~3K planner units.

The one thing it cannot give back is the API's work/reward split: total_micro_usd includes base rewards (model = 'base_reward' rows) while total_count/tokens exclude them. provider_floor_draws — the settlement record behind every base-reward earning — has the per-account amounts but is itself 7.3M rows / 3.2 GB, so summing it per request is not a fast path either (~50 s cold on the replica).

Proposed fix

  1. Add earnings_summary.total_base_reward_micro_usd, maintained by every summary writer.
  2. Backfill it once at boot from provider_floor_draws with the same shape as the existing summary backfill: pinned REPEATABLE READ plan committing per-key deltas with a plan marker, short per-key apply transactions, completion marker.
  3. Answer NetworkTotals(zero since) and Leaderboard(zero since) from the summary (work = total_micro_usd − total_base_reward_micro_usd), joined to the already-indexed reward ledger. Windowed queries unchanged.

Known limitation: base rewards a previous coordinator settles after the plan snapshot during a blue-green overlap are in total_micro_usd but not in the new column — totals stay exact, the split undercounts reward by that amount.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions