feat(stores)!: use backend storage references instead of configs

[ashwinb]

This PR changes configurations in a backward incompatible way.

Run configs today repeat full SQLite/Postgres snippets everywhere a store is needed, which means duplicated credentials, extra connection pools, and lots of drift between files. This PR introduces named storage backends so the stack and providers can share a single catalog and reference those backends by name.

Key Changes

• Add storage.backends to StackRunConfig, register each KV/SQL backend once at startup, and validate that references point to the right family.
• Move server stores under storage.stores with lightweight references (backend + namespace/table) instead of full configs.
• Update every provider/config/doc to use the new reference style; docs/codegen now surface the simplified YAML.

Migration

Before:

metadata_store:
  type: sqlite
  db_path: ~/.llama/distributions/foo/registry.db
inference_store:
  type: postgres
  host: ${env.POSTGRES_HOST}
  port: ${env.POSTGRES_PORT}
  db: ${env.POSTGRES_DB}
  user: ${env.POSTGRES_USER}
  password: ${env.POSTGRES_PASSWORD}
conversations_store:
  type: postgres
  host: ${env.POSTGRES_HOST}
  port: ${env.POSTGRES_PORT}
  db: ${env.POSTGRES_DB}
  user: ${env.POSTGRES_USER}
  password: ${env.POSTGRES_PASSWORD}

After:

storage:
  backends:
    kv_default:
      type: kv_sqlite
      db_path: ~/.llama/distributions/foo/kvstore.db
    sql_default:
      type: sql_postgres
      host: ${env.POSTGRES_HOST}
      port: ${env.POSTGRES_PORT}
      db: ${env.POSTGRES_DB}
      user: ${env.POSTGRES_USER}
      password: ${env.POSTGRES_PASSWORD}
  stores:
    metadata:
      backend: kv_default
      namespace: registry
    inference:
      backend: sql_default
      table_name: inference_store
      max_write_queue_size: 10000
      num_writers: 4
    conversations:
      backend: sql_default
      table_name: openai_conversations

Provider configs follow the same pattern—for example, a Chroma vector adapter switches from:

providers:
  vector_io:
  - provider_id: chromadb
    provider_type: remote::chromadb
    config:
      url: ${env.CHROMADB_URL}
      kvstore:
        type: sqlite
        db_path: ~/.llama/distributions/foo/chroma.db

to:

providers:
  vector_io:
  - provider_id: chromadb
    provider_type: remote::chromadb
    config:
      url: ${env.CHROMADB_URL}
      persistence:
        backend: kv_default
        namespace: vector_io::chroma_remote

Once the backends are declared, everything else just points at them, so rotating credentials or swapping to Postgres happens in one place and the stack reuses a single connection pool.

[cdoern]

I like the direction this is going in, but have a general comment about some class structure here now that it seems these StoreConfig classes are combining functionality more than before.

If this is out of scope of this PR, let's open up a follow up issue to track some of this possible refactoring. Thanks!

[ashwinb]

Thanks for the reviews @leseb, @cdoern... will act on them, and taking PR to draft stage while I clean it up.

[cdoern]

overall this is really nice! just one nit so far.

[ehhuang]

lgtm!

[ashwinb]

ok CI is green. landing!