feat(routing): add account affinity with bounded load spillover

[anupsv]

Summary

Add opt-in, account-scoped provider affinity with deterministic spillover, preserving existing admission and ordinary-routing fallbacks.

• Treat one authenticated account as one user. Rank verified physical machines with rendezvous hashing over account ID, concrete model ID, and canonical inventory identity (with eligible legacy attested serial/SE-key fallback); API-key IDs, request IDs, session IDs, prompts, and live load do not affect the rank.
• Add EIGENINFERENCE_ACCOUNT_AFFINITY_MODE=off|shadow|on (default off) and EIGENINFERENCE_ACCOUNT_AFFINITY_MAX_TTFT_PENALTY_MS (default 250). Off/shadow preserve the existing winner and prefix-affinity tiebreaker.
• In active mode, choose the first ranked warm candidate that passes current admission, projected decode quality, and the request's absolute deadline. The spill bound measures load-induced TTFT above that machine's own estimated idle baseline, not its latency difference from a faster peer: an idle preferred 900 ms machine can remain ahead of a 400 ms alternative.
• Account for whole-machine pending and backend running/waiting work without double-counting overlapping reports. Revalidate identity, load, configuration, and admission before the atomic pending debit.
• Retain at most eight backups, protecting ready candidates from being crowded out by busy identities. Re-evaluate HRW within every reached owner/version/negative-quote priority tier on retries and hedges, including after a higher tier loses admission. Exhaust ordinary-cost fallback within a higher-priority tier before proceeding to a weaker tier. Positive quote arrival does not scramble affinity order.
• Keep identity snapshots and plans request-local: no per-account registry map, timers, or workers. Avoid per-provider identity-string allocations without changing the length-framed hash or identity tie order. Emit only bounded aggregate diagnostics, never account or machine identities.

Before

Ordinary cost and existing prefix-affinity ties determine placement; repeat requests from one account have no dedicated stable machine order. Retry plans retain cost-ranked backups.

flowchart TD
    Request[Authenticated inference request] --> Scan[scanCandidatesLocked + fillRoutingSnapshotPLocked: shared state, hard gates, and pool preferences]
    Scan --> Select[selectRoutingCandidateWithAffinity: ordinary cost and equivalent-prefix tie]
    Select --> Commit{commitProviderReservation: current admission}
    Commit -->|Admitted| Plan[newDispatchPlan: up to 8 cost-ranked backups]
    Plan --> Serve[Dispatch inference and stream response]
    Commit -->|State changed| Scan
    Scan -->|No admissible candidate| Existing[Existing queue, retry, or error outcome]
    Serve -->|Retry or hedge| Retry[ReserveNextFromPlan: quote/version preference, then cost order]
    Retry -->|Current gates pass and debit succeeds| Serve
    Retry -->|Plan exhausted| Refresh[RefreshDispatchPlan: one full rescan]
    Refresh --> Scan

Loading

After

Active affinity adds stable account/model/machine ranking with modest load-based spillover. Higher-priority owner/version groups and all hard admission gates remain authoritative; no extra waiting or affinity-only rejection is introduced.

flowchart TD
    Request[Authenticated inference request] --> Scan[scanCandidatesLocked + fillRoutingSnapshotPLocked: existing gates, shared state, and affinity inputs]
    Scan --> Legacy[selectRoutingCandidateWithAffinity: unchanged legacy baseline]
    Legacy --> Mode{Account affinity mode}
    Mode -->|Off or shadow: retain legacy winner| Commit
    Mode -->|On| Affinity[evaluateAccountAffinity: account/model/verified-machine HRW]
    Affinity -->|First warm candidate within own-load, TPS, and deadline bounds| Commit
    Affinity -->|None feasible: retain legacy winner| Commit
    Commit{commitProviderReservation: recheck identity/load/config and atomic admission} -->|Admitted| Plan[newDispatchPlan + retainEntryBefore: up to 8 ready-first HRW backups]
    Commit -->|State changed| Scan
    Plan --> Serve[Dispatch inference and stream response]
    Serve -->|Retry or hedge| Retry[reserveAccountAffinityPlan: HRW then cost fallback within each owner/version/quote tier]
    Retry --> Guard{accountAffinityPlanGuard + current admission under provider lock}
    Guard -->|Admitted and debited| Serve
    Guard -->|Unavailable| Next{More candidates or priority tiers?}
    Next -->|Yes| Retry
    Next -->|No| Refresh[RefreshDispatchPlan: one full rescan]
    Refresh --> Scan
    Scan -->|No admissible candidate| Existing[Existing queue, retry, or error outcome]

Loading

off and shadow also keep the legacy retry-plan policy. Shadow evaluates primary-choice counterfactuals without serving those alternatives. The active retry path never rescans the fleet until the existing one-shot refresh.

Validation

Passed on local Go 1.27.1, darwin/arm64:

• go test ./registry ./registry/routingsim ./modelpolicy -count=1
• go test ./registry ./api -run '^TestAccountAffinity' -count=1
• go test -race -p 4 ./registry ./api -run 'Test(AccountAffinity|AppAttest|RoutingSnapshot|Offloaded|ReserveCommit|DispatchPlan|ReserveNextFromPlan|RefreshDispatchPlan|PlanFirst|Hedge|Governor|QueueDrain|Drain)' -count=3
• Retry-tier admission-race, higher-priority soft-fallback, and identity hash/tie compatibility regressions were also repeated 50 times before the master integration.
• Full-path allocation regression and BenchmarkAccountAffinityReserve_350x2: off/shadow/on each measure 365 allocs/op on the same 350-provider, two-model attested fixture after merging current master. An untouched archive of d78ae77ef also measures 365 allocs/op with verified identities (13 without them): this is the upstream baseline, not affinity overhead. The regression now requires affinity to add zero allocations above the identical off-mode fleet. Identity hashing avoids per-provider concatenation allocations; no per-account cache or worker lifecycle was added.
• make coordinator-build, go vet ./registry ./api ./cmd/coordinator, make docs-impact-check BASE=origin/master, make docs-check, and git diff --check.

The full coordinator suite (make coordinator-test) is not fully green on this local toolchain. Its only failures are these four existing JSON tests, reproduced on an untouched archive of the updated base d78ae77ef:

• TestJSONStringEncodedLenMatchesEncoder
• TestJSONEncodedLenLeafCases
• TestJSONValueLenMatchesMarshal
• TestCacheBustSpliceMatchesReencode/canonical_raw_line_separators

No live provider-inference E2E or production load experiment was run.

Integration with current master

• Merge d78ae77ef and preserve the new shared fillRoutingSnapshotPLocked helper, native-model offload estimates, and admission budget clamps. Add affinity identity/occupancy to that shared projection, with regression coverage for reused snapshots and disabled mode.
• Support account-bound canonical machine IDs for App Attest-only providers. Unverified serial/key claims on MDM-optional sessions cannot establish affinity; the legacy fallback requires complete MDA-bound serial evidence. A retained identity never bypasses serving-authorization expiry.
• Keep upstream release notes intact and place this unreleased feature in its own section. Update canonical routing, scheduling, API contract, configuration, telemetry, and rollout docs.
• No database migration, provider wire-protocol change, new public request field, or deployment change.

Rollout and limits

• Default remains off. This PR does not change deployment configuration or activate prefix-cache reuse.
• Use the documented dev shadow/on checks and separately approved production rollout. Watch measured tail TTFT, decode TPS, retries/hedges, and 429/timeout rates.
• The own-machine load increment is an estimate, not a wait timer or measured-latency guarantee. It does not reconstruct other models' queued prompt lengths or separate contention already embedded in observed prefill rates.
• Docs cover configuration, routing invariants, telemetry, rollout/rollback, and the distinction between stable placement and actual cache hits.

[vercel]

@anupsv is attempting to deploy a commit to the EigenLabs Team on Vercel.

A member of the Team first needs to authorize it.