Last updated: 2026-09-20 · commit
3b1b6a476
What the coordinator persists, through which interface, in which backend, and
how the schema reaches a fresh database; then what a provider keeps on its own
disk and in its Keychain. Read this to understand what survives a restart, what
does not, and which files an operator may touch. Configuration values are
listed once in ../reference/configuration.md;
the SSD cache file format is in
../reference/ssd-kv-cache.md.
Model versions also store the optional hugging_face_artifact as nullable JSONB
(coordinator/store/postgres.go). SetModelVersion replaces it and invalidates
the existing model read-through cache; artifact schema.
Attempt decision fields are additive columns and existing provider JSONB; prediction telemetry defines migration, historical NULLs and the separately applied waterfall view.
App Attest maintenance marks abandoned pending submissions interrupted without accepting old assertions or changing counters. Enrollment contexts retain their protocol version so an upgrade does not reinterpret a cached proof. Historical receipt recovery appends new versions; credential revocations are durable and account-scoped. See coordinator/store/app_attest_maintenance.go and coordinator/store/app_attest_readiness.go.
The additive App Attest inventory and evidence tables retain stable machine mappings, session/OS history, complete proof bytes, receipt versions, and atomic verification results. They do not restore live authorization. The MDM-optional path uses verified canonical identity for new base-floor settlement, while retaining original organic-earning keys and historical ledger rows.
Machine-session reconciliation uses a partial index over open sessions and bounded, row-locked batches to repair missed disconnect writes after contention or restart. Fresh inventory or provider-session heartbeats preserve liveness. Known closures retain their timestamp; inferred stale closures are labelled and can recover when fresh observations resume. Older observations and confirmed disconnects are fenced in ObserveMachine. See the inventory lifecycle and coordinator/store/machine_inventory_reconcile.go.
Canonical history and reward queries use the existing inventory tables. coordinator/store/machine_inventory_schema.go adds the reverse-merge index darkbloom_machines_merged_into; include it in additive migration preflight. coordinator/store/postgres_machine_floor_settlement.go serializes canonical/raw-session settlement with inventory merges, preventing duplicate same-epoch floors without rewriting existing balances. Serving leases remain in memory and are re-established after reconnect.
Provider attestation_result JSON additively retains OSVersion from the signed
registration blob (coordinator/attestation/attestation.go, VerificationResult).
Existing rows without it decode as unknown. This app-reported metadata supports
owner upgrade notices; it does not certify an OS or alter trust gates, and needs
no SQL migration.
The additive app_attest_build_qualifications table stores immutable signed-artifact approval, test evidence and server-attributed operator/time, plus permanent revocation tombstones. coordinator/store/app_attest_builds_postgres.go (SetQualifiedRelease) locks the same row used for revocation and atomically checks the exact identity before writing the active release. store.As[AppAttestBuildStore] unwraps the store decorator; qualification reads are deliberately uncached there. Service snapshots have a separate bounded lifetime, and stored approval never restores a live serving lease. See the qualification runbook.
The coordinator is a single Go process whose in-memory registry is rebuilt from
provider connections after every restart. Everything that must outlive a
process — accounts and keys, the money ledger, usage, the model catalog,
releases, provider trust decisions and telemetry — goes through one Store
value chosen at boot. Providers are the opposite: a darkbloom daemon keeps a
handful of small JSON files under ~/.darkbloom, its weights in the Hugging
Face cache, an encrypted SSD prefix cache, and one key-encryption key in the
Keychain. Nothing prompt-derived is stored on either side.
Store (coordinator/store/interface.go) is the union of thirteen domain
interfaces declared in coordinator/store/interface_domains.go. Callers depend
on the narrow slice they need; both implementations satisfy all thirteen.
| Sub-interface | Owns |
|---|---|
APIKeyStore |
Consumer API keys: create, seed, validate, per-key limits and counts. |
UsageStore |
Usage events and settled payments plus the totals, time-series, geo and leaderboard aggregations behind /v1/stats. |
RequestOutcomeStore |
Versioned unsampled request_outcomes, unique on coordinator UUID, with bounded compact attempt evidence; see incoming request accounting. |
TelemetryStore |
Routing-decision snapshots (inference_routes), rejection records and the profiler's request_profiles/fleet_snapshots; prompt-free by construction. |
LedgerStore |
The double-entry balance ledger; every amount is micro-USD. |
BillingStore |
Referrals, deposit sessions, per-account model prices and Stripe Connect withdrawals. |
ModelRegistryStore |
The manifest-backed model catalog and the public aliases that resolve to concrete builds. |
ReleaseStore |
Versioned provider binary releases and their hashes. |
UserStore |
Privy-linked consumer accounts, role, platform-fee override and Stripe Connect payout fields. |
DeviceAuthStore |
The RFC 8628-style device-code flow and the long-lived provider tokens it mints. |
InviteStore |
Invite codes and redemptions. |
ProviderEarningsStore |
Per-node earnings, payouts and the base-rewards settlement rows. |
ProviderStore |
Provider records and sessions, reputation, the APNs code-identity and trust-reuse caches, verification jobs and log reports. |
Telemetry events are not in the store at all: TelemetryEventRecord goes to
Datadog only (see telemetry.md).
| Backend | File | Selected when | Durability |
|---|---|---|---|
PostgresStore |
coordinator/store/postgres.go (+ postgres_*.go) |
EIGENINFERENCE_DATABASE_URL is set |
Durable; the only backend for dev and production. In production the database is AWS RDS, outside the coordinator VM and its container, so a container swap or VM reboot cannot touch it (../operations/coordinator-deploy.md). |
MemoryStore |
coordinator/store/memory.go |
No DSN and EIGENINFERENCE_ALLOW_MEMORY_STORE=true |
Process memory; everything is lost on exit. |
Selection is in main (coordinator/cmd/coordinator/main.go): with a DSN it
calls store.NewPostgres and exits 1 on any connect, ping or migration error;
without one it refuses to start unless the memory store is explicitly allowed
(store.Config.Check). The memory store is for tests and local hacking, not a
"simple deployment" — it has no fsync'd trust-reuse journal, no durable code
attestations, and the admin key exists only because store.NewMemory received
it in store.Config.
NewPostgres parses the DSN with pgxpool.ParseConfig and overrides the pool
shape: at least 80 max connections, 10 minimum, 30 minute connection lifetime,
5 minute idle timeout, 30 second health check. The 80 floor exists because the
stats endpoint can hold connections for seconds while heartbeat upserts,
billing settlements and inference completions also need them. The DSN itself is
the only connection-level knob; there is no separate host/user/password set.
coordinator --migrate-only applies the same migrations and exits before admin
key seeding, listeners, or background workers. It requires a PostgreSQL URL;
there is no memory-store fallback or schema-skip mode. Normal startup still
checks every migration. There is no versioned migration directory for the
schema. PostgresStore.migrate (coordinator/store/postgres.go) executes an
ordered slice of idempotent statements — CREATE TABLE IF NOT EXISTS,
ADD COLUMN IF NOT EXISTS, CREATE INDEX IF NOT EXISTS, DROP TABLE IF EXISTS
for retired tables — on every start, followed by:
migrateEarningsSummary (postgres_earnings_summary_migration.go),
ensureProviderRestoreIndexes (postgres_startup.go),
migrateUsageTotals (postgres_usage_totals_migration.go),
migrateWithdrawableBalance (postgres_withdrawable_migration.go) and
ensureProviderEarningsJobIndex. One-shot data migrations are gated by a row
in schema_migrations so they run at most once. Two SQL files under
coordinator/store/migrations/ are deliberately not on that path and are
applied by hand with psql: dedupe_provider_earnings.sql (an offline cleanup
that once ran at boot, held a relation lock for ~15 minutes on the production
table and kept the coordinator from binding its port) and
request_waterfall.sql (an analysis view joining request_profiles to
inference_routes, kept off the boot path so a CREATE VIEW can never queue
behind a long query's lock). coordinator/deploy/start.sh does not touch the
database; it only prepares the persistent disk and MicroMDM before exec coordinator.
The earnings-summary backfill pins a REPEATABLE READ snapshot before publishing
its attempted-plan marker on a separate, bounded database connection. Missing
account and provider keys, and their historical earnings, are read from that
same pinned snapshot. An old writer committing before or after the attempt
marker remains outside the captured history if it committed after the snapshot.
The plan commits its historical deltas to earnings_summary_backfill_pending
alongside the ready-plan marker. Existing counters at the pinned snapshot are
preserved. A short transaction adds one pending key to its live counter and
removes the pending item atomically; the final backfill_earnings_summary_v1
marker is recorded only after the queue is empty.
Retries resume committed progress without rescanning history or double-adding applied keys. Migration runners serialize on a session advisory lock using one leased connection; serving writers do not participate in that lock. Only initial planning opens the extra marker connection; it does not wait for a second pool slot. A failed or uncertain marker write aborts planning. If the attempt marker committed but the ready plan did not, retry refuses to replan against potentially partial counters. The coordinator remains unready until explicit reconciliation; an already committed plan continues automatically from its pending keys.
The existing CreditProviderAccount and SettleProviderFloorDraw SQL statements
insert earnings and update both summaries atomically. A live writer that creates
a counter after the planning snapshot retains its increment when the captured
historical delta is added. Historical scans take no writer-blocking table locks,
and per-key application never holds an account row while waiting on a provider
row. Offline imports or an old record-only writer without atomic summary updates
must be quiesced during preparation. Already inconsistent counters at the
planning snapshot are not recomputed or silently repaired.
RecordProviderEarning and CreditProviderAccount maintain new summaries from
inserted earning rows, so duplicate non-empty job IDs never increment twice.
base_reward contributes money but zero inference count/tokens, matching floor
draw settlement and MemoryStore. The record-only method does not credit balances
or create ledger entries.
Normal upgrades do not require subtracting historical base rewards from existing
summaries: SettleProviderFloorDraw has atomically maintained both summary rows
with zero work since base rewards were introduced, and the former startup
backfill used ON CONFLICT DO NOTHING. Its all-row count therefore did not
overwrite those maintained summaries. Imported earnings or manually deleted and
rebuilt summaries can have different provenance and require evidence-backed
reconciliation. Neither subtracting every retained base reward nor replacing
lifetime counters from retained detail rows is a safe general repair; the
upgrade regression in coordinator/store/earnings_summary_legacy_upgrade_test.go
checks the production sequence and preservation of lifetime totals.
Postgres startup logs connection time and individual schema statement durations as bounded phase labels, plus named backfills/index phases. Logs omit SQL and parameters. The first boot that installs the new marker still performs the historical aggregation; preparing migrations before the drain moves that work out of the cutover window. Concurrent index creation can still wait for old transactions. See the deployment procedure for the approval and compatibility boundary.
Provider history is recovered on demand after successful live SE attestation,
using GetProviderForRestore with the verified serial first, then SE key if
no serial record exists. Ordered partial indexes on each identity plus
last_seen DESC, id DESC select the newest prior session. Every currently registered session is excluded, and a returned row is checked
again for sessions arriving during lookup. Until history lookup/restoration
finishes, persisted rows omit the indexed serial and SE key, including late writes
from a registration that already disconnected. Provider-record and reputation persistence share a mutex, and pending reputation
writes are skipped. Completed records publish together with their reputation in
one Postgres transaction or MemoryStore lock (UpsertProviderWithReputation,
coordinator/store/provider_record_write.go), so an older zero snapshot cannot
overwrite the completed state and no completed identity appears without its
reputation. Registration retries history/reputation reads up to three times,
within one five-second deadline shared with RestoreProviderStateContext.
Verified identities still awaiting restoration fail state_restoring in routing,
capacity and model-loading gates; owner self-route cannot bypass it. Exhaustion
ends the new registration with WebSocket close code 1013 before account lookup,
MDM scheduling or duplicate-session eviction. Normal reconnect can retry against
preserved history. Missing legacy reputation is allowed; a failed read never
marks restoration complete. NewServer does not scan historical providers.
The existing RestoreProviderState trust cap and independent newest-nonempty-MDA
chain re-verification remain in force.
Store errors are logged and do not grant hardware trust. Reputation is still
loaded by the selected historical record ID. ListProviderRecords remains an
explicit administrative store operation and now returns scan/iteration errors
instead of a partial-success list.
flowchart LR
A[ReadAppConfig] --> B{DATABASE_URL set?}
B -- yes --> C[pgxpool connect + Ping]
C --> D[migrate: idempotent DDL slice]
D --> E[one-shot data migrations\ngated by schema_migrations]
E --> F[ensureProviderEarningsJobIndex\nCONCURRENTLY, fast-path if present]
F --> G[SeedKey admin key]
B -- no, ALLOW_MEMORY_STORE=true --> H[NewMemory + 15 min pruner]
B -- no --> X[exit 1]
D -. any error .-> X
Roughly forty tables; grouped by what would be lost if the family vanished.
| Family | Tables | Notes |
|---|---|---|
| Identity and access | api_keys, users, device_codes, provider_tokens, publishing_api_keys, invite_codes, invite_redemptions |
Keys are stored as hashes with a display prefix; users carries the Stripe Connect fields. |
| Money | balances, ledger_entries, billing_sessions, model_prices, referrers, referrals, stripe_withdrawals, global_payout_recipients, global_payout_withdrawals, provider_earnings, earnings_summary, provider_payouts, provider_floor_draws, payments (legacy) |
The ledger is append-only; balances is the materialised view of it. Semantics in billing.md. |
| Usage and routing telemetry | usage, usage_totals, inference_routes, request_rejections, request_profiles, fleet_snapshots, request_outcomes |
Row per request, per dispatched attempt, per rejection, per profiled attempt, per fleet sample; usage_totals is a single-row counter kept by migrateUsageTotals. |
| Provider fleet and trust | providers, provider_reputation, provider_sessions, provider_trust_reuse, provider_verification_jobs, code_attestations, code_attest_push_budgets, provider_log_reports |
Trust reuse and code attestations are durable. code_attestations.continuous_coverage_until is compare-and-updated only for the exact original proof tuple; it never refreshes attested_at or inserts proof. This allows bounded same-process resume after a redeploy; see security/attestation.md. provider_log_reports.serial_number is kept empty by trigger. |
| Models and releases | model_registry, model_versions, model_version_files, model_active_versions, model_aliases, releases |
The catalog the registry syncs at boot; see model-registry.md. |
| Bookkeeping | schema_migrations, earnings_summary_backfill_pending |
Completion/plan markers and resumable per-key historical deltas. |
Claims, result application and definitive-rejection records use one locked PostgreSQL mutation boundary (coordinator/store/global_payouts_postgres.go, mutateGlobalPayout). Operation-specific checks run under the withdrawal row lock; any refund ledger entry and payout update commit together. A no-op claim rolls back without changing the lease or dispatch count.
Payouts marked manual_reconciliation_required without an external payment ID are excluded from automatic scans and claims; their pending row and debit are retained. A verified external ID permits readback reconciliation to resume (coordinator/store/global_payouts.go, GlobalPayout.RequiresManualReconciliation).
Global Payouts uses separate recipient and withdrawal tables with immutable request data, persisted dispatch counts, definitive rejection records, an indexed quote expiry and a unique external-payment index. GlobalPayoutStore is accessed through store.As so decorators preserve the capability. These mutations do not write the cached users table. The migration creates the payout tables and adds/backfills indexed quote expiry for an earlier Global Payouts schema (coordinator/store/global_payouts_postgres.go, globalPayoutSchema). Cleanup locks and removes only expired, never-confirmed quotes in bounded batches; confirmed payout and ledger records are retained (coordinator/store/global_payouts_maintenance.go, PruneExpiredGlobalPayoutQuotes).
Quote invalidation is serialized with confirmation. An invalidation flag prevents an earlier request timestamp from admitting a canceled quote; an already-confirmed payout is returned unchanged for reconciliation (coordinator/store/global_payouts_quote_expiry.go, ExpireGlobalPayoutQuote).
The store keeps most business rows forever; the loops that exist are narrow.
| Loop | Where | What it bounds |
|---|---|---|
| Profiler retention sweep, hourly | coordinator/api/profiler_fleet.go (StartProfilerLoops → PruneTelemetry) |
request_outcomes by receipt time, plus request_profiles and fleet_snapshots older than their retention windows (telemetry-inventory), in batches; runs even when the profiler is off. |
| Memory-store pruner, every 15 minutes | coordinator/cmd/coordinator/main.go (memory_store_pruner, MemoryStore.Prune) |
Append-only history slices to DefaultPruneMaxEntries (100 000); memory store only. |
| Session reconciliation, once at boot | coordinator/cmd/coordinator/main.go (CloseOpenProviderSessions) |
Closes provider_sessions rows whose last heartbeat is more than 3 minutes old, so a blue-green cutover does not truncate live sessions. |
| Read-cache janitor, every minute | coordinator/api/server.go (StartReadCacheJanitor) |
In-process response cache, not a table. |
The existing nullable request_rejections.could_have_served column stores NULL
when counterfactual servability is not evaluated. Go reads it as *bool
(coordinator/store/interface.go, RejectionRecord); both stores preserve
unknown, false, and true. This requires no schema migration.
usage, inference_routes, request_rejections and ledger_entries have no
automatic retention. DeleteExpiredDeviceCodes exists on the interface but has
no scheduled caller.
| What | Where | Owner |
|---|---|---|
| Configuration | ~/.config/darkbloom/provider.toml (legacy ~/.darkbloom/provider.toml is still read) |
provider-swift/Sources/ProviderCore/Config/ProviderConfig.swift |
| Device-auth token | ~/.darkbloom/auth_token |
provider-swift/Sources/ProviderCore/Auth/DeviceAuth.swift |
| Daemon state, PID, warm-model journal, watchdog state, KV-backend crash guard | ~/.darkbloom/daemon-state.json, provider.pid, loaded-models.json, watchdog-state.json, kv-backend-guard.json |
provider-swift/Sources/ProviderCore/Service/ |
| Direct-mode token and discovery file | ~/.darkbloom/local_token, ~/.darkbloom/local.json |
provider-swift/Sources/ProviderCore/Server/LocalEndpoint.swift |
| Logs | ~/.darkbloom/provider.log, ~/.darkbloom/watchdog.log; unified logging via darkbloom logs |
provider-swift/Sources/ProviderCore/Service/LaunchAgent.swift, WatchdogAgent.swift |
| Binaries | ~/.darkbloom/Darkbloom.app, symlinks in ~/.darkbloom/bin/ |
scripts/install.sh |
| Model weights | ~/.cache/huggingface/hub |
provider-swift/Sources/ProviderCore/Models/ModelDownloader.swift |
| SSD prefix cache | ~/Library/Caches/darkbloom/kv3/<model>/ — encrypted blocks; default budget under size and eviction rules |
provider-swift/Sources/ProviderCore/KVCacheSSD/SSDPrefixCacheFactory.swift; format in ../reference/ssd-kv-cache.md |
| KV key-encryption key | Keychain item, service io.darkbloom.kv.kek.v1, access group SLDQ2GJ6TL.io.darkbloom.provider, wrapped by a Secure Enclave key |
provider-swift/Sources/ProviderCore/KVCache/WrappedKEKStorage.swift, Security/PersistentEnclaveKey.swift |
| Fan helper policy | /Library/Application Support/Darkbloom/fan-policy.json, fan-session.json |
../provider/fan-control.md |
Every file path in the first four rows can be moved with the DARKBLOOM_*
variables in ../reference/configuration.md.
Sealed request plaintext never reaches disk on a provider; the SSD cache holds
KV blocks under a per-model key, not tokens.
- A production coordinator never runs on the memory store.
store.Config.Checkfails andmainexits unless a DSN is present or the memory store is opted into by name (coordinator/store/config.go,coordinator/cmd/coordinator/main.go). - Schema changes ship with the binary and are idempotent. Every statement
in
PostgresStore.migratecan run on an already-migrated database; the process serves traffic only after the whole slice succeeds (coordinator/store/postgres.go). - Committed migration progress is not applied twice. Small migrations
commit their marker with their update. Earnings backfill commits its plan,
then each delta with pending-row deletion, before recording completion
(
coordinator/store/postgres_earnings_summary_backfill.go). - Boot never holds a long lock on a hot table. The
provider_earnings(job_id)unique index is builtCONCURRENTLY, only after a duplicate check, and skipped when already valid; the dedupe that violated this lives incoordinator/store/migrations/dedupe_provider_earnings.sqland is manual (ensureProviderEarningsJobIndex). - Money is micro-USD integers in an append-only ledger.
LedgerStoreandbalancesnever store floats; seebilling.md#invariants. - Nothing prompt-derived is persisted.
TelemetryStorerows carry token counts, timings and outcomes only; theserial_numbercolumn ofprovider_log_reportsand the legacycache_affinity_keycolumn are kept empty by triggers (coordinator/store/postgres_log_report_privacy.go,legacyCacheAffinityGuardTriggerincoordinator/store/postgres.go). - Provider secrets never leave the Keychain in the clear. The KV KEK is
wrapped by a Secure Enclave key and the SSD cache is unreadable without it
(
provider-swift/Sources/ProviderCore/KVCache/WrappedKEKStorage.swift).
| Symptom | Cause | Where to look |
|---|---|---|
Coordinator exits 1 at boot with store: run migrations |
A DDL statement failed (permissions, a hand-edited schema, or a CREATE INDEX waiting on a lock) |
The logged statement; pg_stat_activity for blockers. |
Boot fails with an actionable provider_earnings duplicate message |
Rows share a non-empty job_id, so the unique index cannot be built |
Run dedupe_provider_earnings.sql offline, then redeploy. |
EIGENINFERENCE_DATABASE_URL is required in production |
No DSN and no memory-store opt-in | The environment file; see ../operations/coordinator-deploy.md. |
| Billing or key state gone after a restart | The process ran on the memory store | Startup log line using in-memory store. |
/v1/stats slow and pool saturated |
Full scans on usage holding connections; the 80-connection floor is the mitigation, not a fix |
pg_stat_activity; the read cache. |
request_waterfall view missing after a fresh database |
It is applied by hand, not at boot | coordinator/store/migrations/request_waterfall.sql. |
| Provider re-challenged after every coordinator deploy | Trust-reuse rows missing (memory store) or provider_trust_reuse revoked |
security/attestation.md. |
| Provider SSD cache empty after reboot | Budget clamp or block TTL (size and eviction rules), or the KEK item missing | ../reference/ssd-kv-cache.md; darkbloom doctor. |
| Concern | Location |
|---|---|
| Interface and record types | coordinator/store/interface.go, coordinator/store/interface_domains.go |
| Backend selection and validation | coordinator/store/config.go, coordinator/cmd/coordinator/main.go |
| Postgres pool, schema, one-shot migrations | coordinator/store/postgres.go, coordinator/store/postgres_usage_totals_migration.go, coordinator/store/postgres_withdrawable_migration.go, coordinator/store/postgres_log_report_privacy.go |
| Provider identity and usage reads | coordinator/store/postgres_provider_read.go (providerRecordColumns, scanProviderRecord, GetProviderRecord, GetProviderBySerial); coordinator/store/provider_restore.go (GetProviderForRestore, using the same projection); coordinator/store/postgres_usage_read.go (readUsageRecords, UsageRecords, UsageRecordsSince); coordinator/store/postgres_row.go (rowScanner) |
| Domain files | coordinator/store/postgres_model_registry.go, coordinator/store/postgres_base_rewards.go, coordinator/store/postgres_profiles.go, coordinator/store/route_telemetry.go, coordinator/store/usage_time_series.go, coordinator/store/apikey.go |
| Memory backend | coordinator/store/memory.go, coordinator/store/memory_base_rewards.go |
| Manual SQL | coordinator/store/migrations/ |
| Persistent-disk state outside Postgres (MicroMDM, journals) | coordinator/deploy/start.sh, coordinator/api/trust_reuse_journal.go, ../operations/state-export.md |
| Provider files and Keychain | provider-swift/Sources/ProviderCore/Config/ProviderConfig.swift, provider-swift/Sources/ProviderCore/Service/, provider-swift/Sources/ProviderCore/KVCacheSSD/, provider-swift/Sources/ProviderCore/KVCache/WrappedKEKStorage.swift |
../reference/configuration.md—EIGENINFERENCE_DATABASE_URL,EIGENINFERENCE_ALLOW_MEMORY_STORE,USER_PERSISTENT_DATA_PATHand the provider path overridesbilling.md— what the money tables meantelemetry.mdandsystem-profiler.md— what fills the telemetry tablesmodel-registry.md— the catalog tablessecurity/attestation.md— the trust-reuse and code-attestation cachesprefix-cache.mdand../reference/ssd-kv-cache.md— the provider's on-disk cache../operations/state-export.md— exporting the non-Postgres state on the persistent disk../operations/coordinator-deploy.md— where the DSN is set
coordinator/store/model_token_promotions_schema.go (modelTokenPromotionDDL) adds model_token_promotions, account/model keyed model_token_grants, durable model_token_reservations, and provider-account keyed model_token_provider_carries. The carry row retains a sub-micro-dollar payout remainder in [0, 100000000); the additive table also works when upgrading an existing promotion schema. Promotion model IDs deliberately do not reference the model registry, allowing pre-launch setup. The promotion row serializes claims, enforces the maximum claim count and validates the user table’s authoritative account creation timestamp. Account/model uniqueness makes duplicate claims idempotent. Grant counters enforce nonnegative usage/reservations and prevent their sum from exceeding the grant. Reservation terminal state prevents duplicate spending, refunds and provider credit. Claim windows do not expire previously issued grants.
coordinator/store/model_token_settlement_postgres.go (SettleModelTokenReservation) updates grant usage, consumer money, fractional payout carry and provider earnings in one transaction, locking balance rows in account order before the carry row. coordinator/store/model_token_earnings_postgres.go (carryModelTokenEarningPostgres) updates the remainder; the reservation persists the credited whole-micro-dollar payout so replay returns the original result without accumulating fractions again. The optional backend capability is discovered through store.As; grants bypass the user/model read-through caches and no user/model invalidation is needed. Lease recovery is in coordinator/store/model_token_leases.go (ReleaseStaleModelTokenReservations). Operational details: model token promotions.