Skip to content

Latest commit

 

History

History
354 lines (295 loc) · 31.1 KB

File metadata and controls

354 lines (295 loc) · 31.1 KB

Storage

Last updated: 2026-09-20 · commit 3b1b6a476

What the coordinator persists, through which interface, in which backend, and how the schema reaches a fresh database; then what a provider keeps on its own disk and in its Keychain. Read this to understand what survives a restart, what does not, and which files an operator may touch. Configuration values are listed once in ../reference/configuration.md; the SSD cache file format is in ../reference/ssd-kv-cache.md.

Model versions also store the optional hugging_face_artifact as nullable JSONB (coordinator/store/postgres.go). SetModelVersion replaces it and invalidates the existing model read-through cache; artifact schema.

Attempt decision fields are additive columns and existing provider JSONB; prediction telemetry defines migration, historical NULLs and the separately applied waterfall view.

App Attest maintenance marks abandoned pending submissions interrupted without accepting old assertions or changing counters. Enrollment contexts retain their protocol version so an upgrade does not reinterpret a cached proof. Historical receipt recovery appends new versions; credential revocations are durable and account-scoped. See coordinator/store/app_attest_maintenance.go and coordinator/store/app_attest_readiness.go.

The additive App Attest inventory and evidence tables retain stable machine mappings, session/OS history, complete proof bytes, receipt versions, and atomic verification results. They do not restore live authorization. The MDM-optional path uses verified canonical identity for new base-floor settlement, while retaining original organic-earning keys and historical ledger rows.

Machine-session reconciliation uses a partial index over open sessions and bounded, row-locked batches to repair missed disconnect writes after contention or restart. Fresh inventory or provider-session heartbeats preserve liveness. Known closures retain their timestamp; inferred stale closures are labelled and can recover when fresh observations resume. Older observations and confirmed disconnects are fenced in ObserveMachine. See the inventory lifecycle and coordinator/store/machine_inventory_reconcile.go.

Canonical history and reward queries use the existing inventory tables. coordinator/store/machine_inventory_schema.go adds the reverse-merge index darkbloom_machines_merged_into; include it in additive migration preflight. coordinator/store/postgres_machine_floor_settlement.go serializes canonical/raw-session settlement with inventory merges, preventing duplicate same-epoch floors without rewriting existing balances. Serving leases remain in memory and are re-established after reconnect.

Provider attestation_result JSON additively retains OSVersion from the signed registration blob (coordinator/attestation/attestation.go, VerificationResult). Existing rows without it decode as unknown. This app-reported metadata supports owner upgrade notices; it does not certify an OS or alter trust gates, and needs no SQL migration.

The additive app_attest_build_qualifications table stores immutable signed-artifact approval, test evidence and server-attributed operator/time, plus permanent revocation tombstones. coordinator/store/app_attest_builds_postgres.go (SetQualifiedRelease) locks the same row used for revocation and atomically checks the exact identity before writing the active release. store.As[AppAttestBuildStore] unwraps the store decorator; qualification reads are deliberately uncached there. Service snapshots have a separate bounded lifetime, and stored approval never restores a live serving lease. See the qualification runbook.

Context

The coordinator is a single Go process whose in-memory registry is rebuilt from provider connections after every restart. Everything that must outlive a process — accounts and keys, the money ledger, usage, the model catalog, releases, provider trust decisions and telemetry — goes through one Store value chosen at boot. Providers are the opposite: a darkbloom daemon keeps a handful of small JSON files under ~/.darkbloom, its weights in the Hugging Face cache, an encrypted SSD prefix cache, and one key-encryption key in the Keychain. Nothing prompt-derived is stored on either side.

Mechanism

The store interface

Store (coordinator/store/interface.go) is the union of thirteen domain interfaces declared in coordinator/store/interface_domains.go. Callers depend on the narrow slice they need; both implementations satisfy all thirteen.

Sub-interface Owns
APIKeyStore Consumer API keys: create, seed, validate, per-key limits and counts.
UsageStore Usage events and settled payments plus the totals, time-series, geo and leaderboard aggregations behind /v1/stats.
RequestOutcomeStore Versioned unsampled request_outcomes, unique on coordinator UUID, with bounded compact attempt evidence; see incoming request accounting.
TelemetryStore Routing-decision snapshots (inference_routes), rejection records and the profiler's request_profiles/fleet_snapshots; prompt-free by construction.
LedgerStore The double-entry balance ledger; every amount is micro-USD.
BillingStore Referrals, deposit sessions, per-account model prices and Stripe Connect withdrawals.
ModelRegistryStore The manifest-backed model catalog and the public aliases that resolve to concrete builds.
ReleaseStore Versioned provider binary releases and their hashes.
UserStore Privy-linked consumer accounts, role, platform-fee override and Stripe Connect payout fields.
DeviceAuthStore The RFC 8628-style device-code flow and the long-lived provider tokens it mints.
InviteStore Invite codes and redemptions.
ProviderEarningsStore Per-node earnings, payouts and the base-rewards settlement rows.
ProviderStore Provider records and sessions, reputation, the APNs code-identity and trust-reuse caches, verification jobs and log reports.

Telemetry events are not in the store at all: TelemetryEventRecord goes to Datadog only (see telemetry.md).

Two implementations and when each runs

Backend File Selected when Durability
PostgresStore coordinator/store/postgres.go (+ postgres_*.go) EIGENINFERENCE_DATABASE_URL is set Durable; the only backend for dev and production. In production the database is AWS RDS, outside the coordinator VM and its container, so a container swap or VM reboot cannot touch it (../operations/coordinator-deploy.md).
MemoryStore coordinator/store/memory.go No DSN and EIGENINFERENCE_ALLOW_MEMORY_STORE=true Process memory; everything is lost on exit.

Selection is in main (coordinator/cmd/coordinator/main.go): with a DSN it calls store.NewPostgres and exits 1 on any connect, ping or migration error; without one it refuses to start unless the memory store is explicitly allowed (store.Config.Check). The memory store is for tests and local hacking, not a "simple deployment" — it has no fsync'd trust-reuse journal, no durable code attestations, and the admin key exists only because store.NewMemory received it in store.Config.

Connecting to Postgres

NewPostgres parses the DSN with pgxpool.ParseConfig and overrides the pool shape: at least 80 max connections, 10 minimum, 30 minute connection lifetime, 5 minute idle timeout, 30 second health check. The 80 floor exists because the stats endpoint can hold connections for seconds while heartbeat upserts, billing settlements and inference completions also need them. The DSN itself is the only connection-level knob; there is no separate host/user/password set.

Migrations run inside the process, at every boot

coordinator --migrate-only applies the same migrations and exits before admin key seeding, listeners, or background workers. It requires a PostgreSQL URL; there is no memory-store fallback or schema-skip mode. Normal startup still checks every migration. There is no versioned migration directory for the schema. PostgresStore.migrate (coordinator/store/postgres.go) executes an ordered slice of idempotent statements — CREATE TABLE IF NOT EXISTS, ADD COLUMN IF NOT EXISTS, CREATE INDEX IF NOT EXISTS, DROP TABLE IF EXISTS for retired tables — on every start, followed by: migrateEarningsSummary (postgres_earnings_summary_migration.go), ensureProviderRestoreIndexes (postgres_startup.go), migrateUsageTotals (postgres_usage_totals_migration.go), migrateWithdrawableBalance (postgres_withdrawable_migration.go) and ensureProviderEarningsJobIndex. One-shot data migrations are gated by a row in schema_migrations so they run at most once. Two SQL files under coordinator/store/migrations/ are deliberately not on that path and are applied by hand with psql: dedupe_provider_earnings.sql (an offline cleanup that once ran at boot, held a relation lock for ~15 minutes on the production table and kept the coordinator from binding its port) and request_waterfall.sql (an analysis view joining request_profiles to inference_routes, kept off the boot path so a CREATE VIEW can never queue behind a long query's lock). coordinator/deploy/start.sh does not touch the database; it only prepares the persistent disk and MicroMDM before exec coordinator.

The earnings-summary backfill pins a REPEATABLE READ snapshot before publishing its attempted-plan marker on a separate, bounded database connection. Missing account and provider keys, and their historical earnings, are read from that same pinned snapshot. An old writer committing before or after the attempt marker remains outside the captured history if it committed after the snapshot. The plan commits its historical deltas to earnings_summary_backfill_pending alongside the ready-plan marker. Existing counters at the pinned snapshot are preserved. A short transaction adds one pending key to its live counter and removes the pending item atomically; the final backfill_earnings_summary_v1 marker is recorded only after the queue is empty.

Retries resume committed progress without rescanning history or double-adding applied keys. Migration runners serialize on a session advisory lock using one leased connection; serving writers do not participate in that lock. Only initial planning opens the extra marker connection; it does not wait for a second pool slot. A failed or uncertain marker write aborts planning. If the attempt marker committed but the ready plan did not, retry refuses to replan against potentially partial counters. The coordinator remains unready until explicit reconciliation; an already committed plan continues automatically from its pending keys.

The existing CreditProviderAccount and SettleProviderFloorDraw SQL statements insert earnings and update both summaries atomically. A live writer that creates a counter after the planning snapshot retains its increment when the captured historical delta is added. Historical scans take no writer-blocking table locks, and per-key application never holds an account row while waiting on a provider row. Offline imports or an old record-only writer without atomic summary updates must be quiesced during preparation. Already inconsistent counters at the planning snapshot are not recomputed or silently repaired.

RecordProviderEarning and CreditProviderAccount maintain new summaries from inserted earning rows, so duplicate non-empty job IDs never increment twice. base_reward contributes money but zero inference count/tokens, matching floor draw settlement and MemoryStore. The record-only method does not credit balances or create ledger entries.

Normal upgrades do not require subtracting historical base rewards from existing summaries: SettleProviderFloorDraw has atomically maintained both summary rows with zero work since base rewards were introduced, and the former startup backfill used ON CONFLICT DO NOTHING. Its all-row count therefore did not overwrite those maintained summaries. Imported earnings or manually deleted and rebuilt summaries can have different provenance and require evidence-backed reconciliation. Neither subtracting every retained base reward nor replacing lifetime counters from retained detail rows is a safe general repair; the upgrade regression in coordinator/store/earnings_summary_legacy_upgrade_test.go checks the production sequence and preservation of lifetime totals.

Postgres startup logs connection time and individual schema statement durations as bounded phase labels, plus named backfills/index phases. Logs omit SQL and parameters. The first boot that installs the new marker still performs the historical aggregation; preparing migrations before the drain moves that work out of the cutover window. Concurrent index creation can still wait for old transactions. See the deployment procedure for the approval and compatibility boundary.

Provider history is recovered on demand after successful live SE attestation, using GetProviderForRestore with the verified serial first, then SE key if no serial record exists. Ordered partial indexes on each identity plus last_seen DESC, id DESC select the newest prior session. Every currently registered session is excluded, and a returned row is checked again for sessions arriving during lookup. Until history lookup/restoration finishes, persisted rows omit the indexed serial and SE key, including late writes from a registration that already disconnected. Provider-record and reputation persistence share a mutex, and pending reputation writes are skipped. Completed records publish together with their reputation in one Postgres transaction or MemoryStore lock (UpsertProviderWithReputation, coordinator/store/provider_record_write.go), so an older zero snapshot cannot overwrite the completed state and no completed identity appears without its reputation. Registration retries history/reputation reads up to three times, within one five-second deadline shared with RestoreProviderStateContext. Verified identities still awaiting restoration fail state_restoring in routing, capacity and model-loading gates; owner self-route cannot bypass it. Exhaustion ends the new registration with WebSocket close code 1013 before account lookup, MDM scheduling or duplicate-session eviction. Normal reconnect can retry against preserved history. Missing legacy reputation is allowed; a failed read never marks restoration complete. NewServer does not scan historical providers. The existing RestoreProviderState trust cap and independent newest-nonempty-MDA chain re-verification remain in force. Store errors are logged and do not grant hardware trust. Reputation is still loaded by the selected historical record ID. ListProviderRecords remains an explicit administrative store operation and now returns scan/iteration errors instead of a partial-success list.

flowchart LR
  A[ReadAppConfig] --> B{DATABASE_URL set?}
  B -- yes --> C[pgxpool connect + Ping]
  C --> D[migrate: idempotent DDL slice]
  D --> E[one-shot data migrations\ngated by schema_migrations]
  E --> F[ensureProviderEarningsJobIndex\nCONCURRENTLY, fast-path if present]
  F --> G[SeedKey admin key]
  B -- no, ALLOW_MEMORY_STORE=true --> H[NewMemory + 15 min pruner]
  B -- no --> X[exit 1]
  D -. any error .-> X
Loading

Table families

Roughly forty tables; grouped by what would be lost if the family vanished.

Family Tables Notes
Identity and access api_keys, users, device_codes, provider_tokens, publishing_api_keys, invite_codes, invite_redemptions Keys are stored as hashes with a display prefix; users carries the Stripe Connect fields.
Money balances, ledger_entries, billing_sessions, model_prices, referrers, referrals, stripe_withdrawals, global_payout_recipients, global_payout_withdrawals, provider_earnings, earnings_summary, provider_payouts, provider_floor_draws, payments (legacy) The ledger is append-only; balances is the materialised view of it. Semantics in billing.md.
Usage and routing telemetry usage, usage_totals, inference_routes, request_rejections, request_profiles, fleet_snapshots, request_outcomes Row per request, per dispatched attempt, per rejection, per profiled attempt, per fleet sample; usage_totals is a single-row counter kept by migrateUsageTotals.
Provider fleet and trust providers, provider_reputation, provider_sessions, provider_trust_reuse, provider_verification_jobs, code_attestations, code_attest_push_budgets, provider_log_reports Trust reuse and code attestations are durable. code_attestations.continuous_coverage_until is compare-and-updated only for the exact original proof tuple; it never refreshes attested_at or inserts proof. This allows bounded same-process resume after a redeploy; see security/attestation.md. provider_log_reports.serial_number is kept empty by trigger.
Models and releases model_registry, model_versions, model_version_files, model_active_versions, model_aliases, releases The catalog the registry syncs at boot; see model-registry.md.
Bookkeeping schema_migrations, earnings_summary_backfill_pending Completion/plan markers and resumable per-key historical deltas.

Global Payouts state

Claims, result application and definitive-rejection records use one locked PostgreSQL mutation boundary (coordinator/store/global_payouts_postgres.go, mutateGlobalPayout). Operation-specific checks run under the withdrawal row lock; any refund ledger entry and payout update commit together. A no-op claim rolls back without changing the lease or dispatch count.

Payouts marked manual_reconciliation_required without an external payment ID are excluded from automatic scans and claims; their pending row and debit are retained. A verified external ID permits readback reconciliation to resume (coordinator/store/global_payouts.go, GlobalPayout.RequiresManualReconciliation).

Global Payouts uses separate recipient and withdrawal tables with immutable request data, persisted dispatch counts, definitive rejection records, an indexed quote expiry and a unique external-payment index. GlobalPayoutStore is accessed through store.As so decorators preserve the capability. These mutations do not write the cached users table. The migration creates the payout tables and adds/backfills indexed quote expiry for an earlier Global Payouts schema (coordinator/store/global_payouts_postgres.go, globalPayoutSchema). Cleanup locks and removes only expired, never-confirmed quotes in bounded batches; confirmed payout and ledger records are retained (coordinator/store/global_payouts_maintenance.go, PruneExpiredGlobalPayoutQuotes).

Quote invalidation is serialized with confirmation. An invalidation flag prevents an earlier request timestamp from admitting a canceled quote; an already-confirmed payout is returned unchanged for reconciliation (coordinator/store/global_payouts_quote_expiry.go, ExpireGlobalPayoutQuote).

Retention and pruning

The store keeps most business rows forever; the loops that exist are narrow.

Loop Where What it bounds
Profiler retention sweep, hourly coordinator/api/profiler_fleet.go (StartProfilerLoopsPruneTelemetry) request_outcomes by receipt time, plus request_profiles and fleet_snapshots older than their retention windows (telemetry-inventory), in batches; runs even when the profiler is off.
Memory-store pruner, every 15 minutes coordinator/cmd/coordinator/main.go (memory_store_pruner, MemoryStore.Prune) Append-only history slices to DefaultPruneMaxEntries (100 000); memory store only.
Session reconciliation, once at boot coordinator/cmd/coordinator/main.go (CloseOpenProviderSessions) Closes provider_sessions rows whose last heartbeat is more than 3 minutes old, so a blue-green cutover does not truncate live sessions.
Read-cache janitor, every minute coordinator/api/server.go (StartReadCacheJanitor) In-process response cache, not a table.

The existing nullable request_rejections.could_have_served column stores NULL when counterfactual servability is not evaluated. Go reads it as *bool (coordinator/store/interface.go, RejectionRecord); both stores preserve unknown, false, and true. This requires no schema migration.

usage, inference_routes, request_rejections and ledger_entries have no automatic retention. DeleteExpiredDeviceCodes exists on the interface but has no scheduled caller.

Provider-side local storage

What Where Owner
Configuration ~/.config/darkbloom/provider.toml (legacy ~/.darkbloom/provider.toml is still read) provider-swift/Sources/ProviderCore/Config/ProviderConfig.swift
Device-auth token ~/.darkbloom/auth_token provider-swift/Sources/ProviderCore/Auth/DeviceAuth.swift
Daemon state, PID, warm-model journal, watchdog state, KV-backend crash guard ~/.darkbloom/daemon-state.json, provider.pid, loaded-models.json, watchdog-state.json, kv-backend-guard.json provider-swift/Sources/ProviderCore/Service/
Direct-mode token and discovery file ~/.darkbloom/local_token, ~/.darkbloom/local.json provider-swift/Sources/ProviderCore/Server/LocalEndpoint.swift
Logs ~/.darkbloom/provider.log, ~/.darkbloom/watchdog.log; unified logging via darkbloom logs provider-swift/Sources/ProviderCore/Service/LaunchAgent.swift, WatchdogAgent.swift
Binaries ~/.darkbloom/Darkbloom.app, symlinks in ~/.darkbloom/bin/ scripts/install.sh
Model weights ~/.cache/huggingface/hub provider-swift/Sources/ProviderCore/Models/ModelDownloader.swift
SSD prefix cache ~/Library/Caches/darkbloom/kv3/<model>/ — encrypted blocks; default budget under size and eviction rules provider-swift/Sources/ProviderCore/KVCacheSSD/SSDPrefixCacheFactory.swift; format in ../reference/ssd-kv-cache.md
KV key-encryption key Keychain item, service io.darkbloom.kv.kek.v1, access group SLDQ2GJ6TL.io.darkbloom.provider, wrapped by a Secure Enclave key provider-swift/Sources/ProviderCore/KVCache/WrappedKEKStorage.swift, Security/PersistentEnclaveKey.swift
Fan helper policy /Library/Application Support/Darkbloom/fan-policy.json, fan-session.json ../provider/fan-control.md

Every file path in the first four rows can be moved with the DARKBLOOM_* variables in ../reference/configuration.md. Sealed request plaintext never reaches disk on a provider; the SSD cache holds KV blocks under a per-model key, not tokens.

Invariants

  1. A production coordinator never runs on the memory store. store.Config.Check fails and main exits unless a DSN is present or the memory store is opted into by name (coordinator/store/config.go, coordinator/cmd/coordinator/main.go).
  2. Schema changes ship with the binary and are idempotent. Every statement in PostgresStore.migrate can run on an already-migrated database; the process serves traffic only after the whole slice succeeds (coordinator/store/postgres.go).
  3. Committed migration progress is not applied twice. Small migrations commit their marker with their update. Earnings backfill commits its plan, then each delta with pending-row deletion, before recording completion (coordinator/store/postgres_earnings_summary_backfill.go).
  4. Boot never holds a long lock on a hot table. The provider_earnings(job_id) unique index is built CONCURRENTLY, only after a duplicate check, and skipped when already valid; the dedupe that violated this lives in coordinator/store/migrations/dedupe_provider_earnings.sql and is manual (ensureProviderEarningsJobIndex).
  5. Money is micro-USD integers in an append-only ledger. LedgerStore and balances never store floats; see billing.md#invariants.
  6. Nothing prompt-derived is persisted. TelemetryStore rows carry token counts, timings and outcomes only; the serial_number column of provider_log_reports and the legacy cache_affinity_key column are kept empty by triggers (coordinator/store/postgres_log_report_privacy.go, legacyCacheAffinityGuardTrigger in coordinator/store/postgres.go).
  7. Provider secrets never leave the Keychain in the clear. The KV KEK is wrapped by a Secure Enclave key and the SSD cache is unreadable without it (provider-swift/Sources/ProviderCore/KVCache/WrappedKEKStorage.swift).

Failure modes

Symptom Cause Where to look
Coordinator exits 1 at boot with store: run migrations A DDL statement failed (permissions, a hand-edited schema, or a CREATE INDEX waiting on a lock) The logged statement; pg_stat_activity for blockers.
Boot fails with an actionable provider_earnings duplicate message Rows share a non-empty job_id, so the unique index cannot be built Run dedupe_provider_earnings.sql offline, then redeploy.
EIGENINFERENCE_DATABASE_URL is required in production No DSN and no memory-store opt-in The environment file; see ../operations/coordinator-deploy.md.
Billing or key state gone after a restart The process ran on the memory store Startup log line using in-memory store.
/v1/stats slow and pool saturated Full scans on usage holding connections; the 80-connection floor is the mitigation, not a fix pg_stat_activity; the read cache.
request_waterfall view missing after a fresh database It is applied by hand, not at boot coordinator/store/migrations/request_waterfall.sql.
Provider re-challenged after every coordinator deploy Trust-reuse rows missing (memory store) or provider_trust_reuse revoked security/attestation.md.
Provider SSD cache empty after reboot Budget clamp or block TTL (size and eviction rules), or the KEK item missing ../reference/ssd-kv-cache.md; darkbloom doctor.

Code map

Concern Location
Interface and record types coordinator/store/interface.go, coordinator/store/interface_domains.go
Backend selection and validation coordinator/store/config.go, coordinator/cmd/coordinator/main.go
Postgres pool, schema, one-shot migrations coordinator/store/postgres.go, coordinator/store/postgres_usage_totals_migration.go, coordinator/store/postgres_withdrawable_migration.go, coordinator/store/postgres_log_report_privacy.go
Provider identity and usage reads coordinator/store/postgres_provider_read.go (providerRecordColumns, scanProviderRecord, GetProviderRecord, GetProviderBySerial); coordinator/store/provider_restore.go (GetProviderForRestore, using the same projection); coordinator/store/postgres_usage_read.go (readUsageRecords, UsageRecords, UsageRecordsSince); coordinator/store/postgres_row.go (rowScanner)
Domain files coordinator/store/postgres_model_registry.go, coordinator/store/postgres_base_rewards.go, coordinator/store/postgres_profiles.go, coordinator/store/route_telemetry.go, coordinator/store/usage_time_series.go, coordinator/store/apikey.go
Memory backend coordinator/store/memory.go, coordinator/store/memory_base_rewards.go
Manual SQL coordinator/store/migrations/
Persistent-disk state outside Postgres (MicroMDM, journals) coordinator/deploy/start.sh, coordinator/api/trust_reuse_journal.go, ../operations/state-export.md
Provider files and Keychain provider-swift/Sources/ProviderCore/Config/ProviderConfig.swift, provider-swift/Sources/ProviderCore/Service/, provider-swift/Sources/ProviderCore/KVCacheSSD/, provider-swift/Sources/ProviderCore/KVCache/WrappedKEKStorage.swift

Related

Model token grants

coordinator/store/model_token_promotions_schema.go (modelTokenPromotionDDL) adds model_token_promotions, account/model keyed model_token_grants, durable model_token_reservations, and provider-account keyed model_token_provider_carries. The carry row retains a sub-micro-dollar payout remainder in [0, 100000000); the additive table also works when upgrading an existing promotion schema. Promotion model IDs deliberately do not reference the model registry, allowing pre-launch setup. The promotion row serializes claims, enforces the maximum claim count and validates the user table’s authoritative account creation timestamp. Account/model uniqueness makes duplicate claims idempotent. Grant counters enforce nonnegative usage/reservations and prevent their sum from exceeding the grant. Reservation terminal state prevents duplicate spending, refunds and provider credit. Claim windows do not expire previously issued grants.

coordinator/store/model_token_settlement_postgres.go (SettleModelTokenReservation) updates grant usage, consumer money, fractional payout carry and provider earnings in one transaction, locking balance rows in account order before the carry row. coordinator/store/model_token_earnings_postgres.go (carryModelTokenEarningPostgres) updates the remainder; the reservation persists the credited whole-micro-dollar payout so replay returns the original result without accumulating fractions again. The optional backend capability is discovered through store.As; grants bypass the user/model read-through caches and no user/model invalidation is needed. Lease recovery is in coordinator/store/model_token_leases.go (ReleaseStaleModelTokenReservations). Operational details: model token promotions.