chore: bump v0.16.0 (#3095)

Author: carenthomas

.skills/db-migrations-schema-changes/SKILL.md

@@ -0,0 +1,111 @@
+---
+name: DB migrations and schema changes
+description: >-
+  Workflows and commands for managing Alembic database migrations and schema changes
+  in the letta-cloud core app, including using uv, just, LETTA_PG_URI, and
+  switching between SQLite and Postgres.
+---
+
+# DB migrations and schema changes (letta-cloud core)
+
+Use this skill whenever you need to change the database schema or debug Alembic
+migrations in `apps/core` of the letta-cloud repo.
+
+This skill assumes:
+- Working directory: `apps/core`
+- Migrations: Alembic in `apps/core/alembic`
+- Python runner: `uv`
+- Helper: `just ready` for environment + DB setup
+
+## Quick start
+
+1. Ensure environment is ready:
+   - `just ready`
+2. For Postgres migrations, set:
+   - `export LETTA_PG_URI=postgresql+pg8000://postgres:postgres@localhost:5432/letta-core`
+3. Make your ORM/schema change.
+4. Autogenerate migration:
+   - `uv run alembic revision --autogenerate -m "<short_message>"`
+5. Apply migration:
+   - `uv run alembic upgrade head`
+
+See `references/migration-commands.md` for exact commands and variants.
+
+## Standard workflows
+
+### 1. Add or modify a column (ORM-first)
+
+1. Identify the ORM model and table.
+2. Update the SQLAlchemy model in `letta/orm/...`:
+   - Prefer using mixins (e.g. `ProjectMixin`) when available instead of
+     duplicating columns.
+3. Run `just ready` if dependencies or environment may have changed.
+4. Ensure `LETTA_PG_URI` is set if you want the migration to target Postgres.
+5. Autogenerate Alembic revision with `uv`.
+6. Inspect the generated file under `alembic/versions/`:
+   - Confirm `op.add_column` / `op.alter_column` match expectations.
+7. Apply migrations with `uv run alembic upgrade head`.
+
+Use this pattern for changes like adding `project_id` columns via `ProjectMixin`.
+
+### 2. Data backfill / one-off data migration
+
+1. Make sure the schema change (if any) is already represented in ORM + Alembic.
+2. Create a new Alembic revision **without** autogenerate (or edit an
+   autogen file) and add Python logic in `upgrade()` that:
+   - Uses `op.get_bind()` and SQLAlchemy Core/SQL to backfill data.
+3. Keep `downgrade()` simple and safe (ideally reversible).
+4. Run against Postgres with `LETTA_PG_URI` set, using `uv run alembic upgrade head`.
+
+### 3. Fixing a bad migration
+
+Typical cases:
+- Migration fails only on SQLite (ALTER constraint limitations).
+- Migration was generated while pointing at SQLite instead of Postgres.
+
+Workflow:
+1. Identify the failing revision in `alembic/versions/`.
+2. If failure is SQLite-specific, prefer running migrations against Postgres by
+   exporting `LETTA_PG_URI` and re-running upgrade.
+3. If logic is wrong, create a **new** migration that fixes the problem rather
+   than editing an applied revision (especially in shared environments).
+4. For purely local/dev history, you can delete and regenerate migrations but
+   only if no one else depends on them.
+
+See `references/sqlite-vs-postgres-gotchas.md` for SQLite-specific issues.
+
+### 4. Switching between SQLite and Postgres
+
+Alembic picks the engine based on `letta.settings.DatabaseChoice` and
+environment variables.
+
+General rules:
+- For local dev stateful runs, `just ready` handles baseline migrations.
+- For schema design and production-like migrations, prefer Postgres and set
+  `LETTA_PG_URI`.
+
+Workflow for Postgres-targeted migration:
+1. `export LETTA_PG_URI=postgresql+pg8000://postgres:postgres@localhost:5432/letta-core`
+2. From `apps/core`:
+   - `uv run alembic upgrade head`
+   - `uv run alembic revision --autogenerate -m "..."`
+
+## Troubleshooting
+
+- **"Target database is not up to date" when autogenerating**
+  - First run `uv run alembic upgrade head` (with appropriate engine/URI).
+- **SQLite NotImplementedError about ALTER CONSTRAINT**
+  - Switch to Postgres by setting `LETTA_PG_URI` and rerun.
+- **Autogenerated migration missing expected changes**
+  - Ensure ORM imports and metadata (`Base.metadata`) are correct and that the
+    changed model is imported in Alembic env context.
+- **Autogenerated migration has unexpected drops/renames**
+  - Review model changes; consider explicit operations instead of relying on
+    autogenerate.
+
+## References
+
+- `references/migration-commands.md` — canonical commands for `uv`, Alembic,
+  and `just`.
+- `references/sqlite-vs-postgres-gotchas.md` — engine-specific pitfalls and
+  how to avoid them.

.skills/db-migrations-schema-changes/references/migration-commands.md

@@ -0,0 +1,72 @@
+# Migration commands (letta-cloud core)
+
+Working directory for all commands: `apps/core`.
+
+## Environment setup
+
+- One-shot environment + DB setup:
+  - `just ready`
+
+## Postgres connection
+
+Set the Postgres URI (adjust as needed for your env):
+
+```bash
+export LETTA_PG_URI=postgresql+pg8000://postgres:postgres@localhost:5432/letta-core
+```
+
+Alembic will log the effective URL (e.g. `Using database:  postgresql+pg8000://...`).
+
+## Alembic basics (with uv)
+
+- Upgrade to latest:
+
+```bash
+uv run alembic upgrade head
+```
+
+- Downgrade one step:
+
+```bash
+uv run alembic downgrade -1
+```
+
+- Downgrade to a specific revision:
+
+```bash
+uv run alembic downgrade <revision_id>
+```
+
+- Generate new revision (autogenerate):
+
+```bash
+uv run alembic revision --autogenerate -m "short_message"
+```
+
+- Generate empty revision (manual operations):
+
+```bash
+uv run alembic revision -m "manual_migration"
+```
+
+## Typical workflow snippets
+
+### Add/modify column
+
+```bash
+cd apps/core
+just ready                       # optional but recommended
+export LETTA_PG_URI=postgresql+pg8000://postgres:postgres@localhost:5432/letta-core
+uv run alembic upgrade head      # ensure DB is up to date
+uv run alembic revision --autogenerate -m "add_<column>_to_<table>"
+uv run alembic upgrade head
+```
+
+### Re-run last migration after edit (local only)
+
+```bash
+cd apps/core
+export LETTA_PG_URI=postgresql+pg8000://postgres:postgres@localhost:5432/letta-core
+uv run alembic downgrade -1
+uv run alembic upgrade head
+```

.skills/db-migrations-schema-changes/references/sqlite-vs-postgres-gotchas.md

@@ -0,0 +1,41 @@
+# SQLite vs Postgres gotchas (letta-cloud core)
+
+## SQLite limitations
+
+SQLite has limited support for `ALTER TABLE`, especially around constraints and
+foreign keys. In Alembic this often shows up as:
+
+- `NotImplementedError: No support for ALTER of constraints in SQLite dialect...`
+
+In `apps/core`, you may hit this when running migrations against SQLite that
+drop or change foreign keys or constraints.
+
+### How to handle
+
+- Prefer running schema-changing migrations against Postgres by setting
+  `LETTA_PG_URI` and using:
+
+  ```bash
+  uv run alembic upgrade head
+  ```
+
+- If you must support SQLite, use Alembic batch mode patterns, but for this
+  project most complex migrations should target Postgres.
+
+## Autogenerate differences
+
+Running `alembic revision --autogenerate` against SQLite vs Postgres can
+produce different diffs (especially around indexes and constraints).
+
+Recommendations:
+
+- For production/real migrations, always autogenerate against Postgres.
+- If you see lots of unexpected index drops/adds, confirm which engine the
+  migration is inspecting and rerun with Postgres.
+
+## Engine selection reminder
+
+- Engine is controlled by `letta.settings.DatabaseChoice` and environment
+  (notably `LETTA_PG_URI`).
+- `just ready` will also run migrations; ensure your desired engine is set
+  before relying on its database steps.

alembic/versions/175dd10fb916_add_prompt_tokens_details_to_steps.py

@@ -0,0 +1,29 @@
+"""Add prompt_tokens_details to steps table
+
+Revision ID: 175dd10fb916
+Revises: b1c2d3e4f5a6
+Create Date: 2025-11-28 12:00:00.000000
+
+"""
+
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision: str = "175dd10fb916"
+down_revision: Union[str, None] = "b1c2d3e4f5a6"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    # Add prompt_tokens_details JSON column to steps table
+    # This stores detailed prompt token breakdown (cached_tokens, cache_read_tokens, cache_creation_tokens)
+    op.add_column("steps", sa.Column("prompt_tokens_details", sa.JSON(), nullable=True))
+
+
+def downgrade() -> None:
+    op.drop_column("steps", "prompt_tokens_details")

alembic/versions/2e5e90d3cdf8_add_project_id_to_tools.py

@@ -0,0 +1,27 @@
+"""add project_id to tools
+
+Revision ID: 2e5e90d3cdf8
+Revises: af842aa6f743
+Create Date: 2025-12-03 11:55:57.355341
+
+"""
+
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision: str = "2e5e90d3cdf8"
+down_revision: Union[str, None] = "af842aa6f743"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column("tools", sa.Column("project_id", sa.String(), nullable=True))
+
+
+def downgrade() -> None:
+    op.drop_column("tools", "project_id")

alembic/versions/af842aa6f743_add_tool_indexes_for_organization_id.py

@@ -0,0 +1,33 @@
+"""add tool indexes for organization_id
+
+Revision ID: af842aa6f743
+Revises: 175dd10fb916
+Create Date: 2025-12-07 15:30:43.407495
+
+"""
+
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision: str = "af842aa6f743"
+down_revision: Union[str, None] = "175dd10fb916"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_index("ix_tools_organization_id", "tools", ["organization_id"], unique=False)
+    op.create_index("ix_tools_organization_id_name", "tools", ["organization_id", "name"], unique=False)
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_index(op.f("ix_step_metrics_run_id"), "step_metrics", ["run_id"], unique=False)
+    op.create_index(op.f("idx_messages_step_id"), "messages", ["step_id"], unique=False)
+    # ### end Alembic commands ###

alembic/versions/d0880aae6cee_add_compaction_settings_to_agents_table.py

@@ -0,0 +1,28 @@
+"""add compaction_settings to agents table
+
+Revision ID: d0880aae6cee
+Revises: 2e5e90d3cdf8
+Create Date: 2025-12-10 16:17:23.595775
+
+"""
+
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+
+from alembic import op
+from letta.orm.custom_columns import CompactionSettingsColumn
+
+# revision identifiers, used by Alembic.
+revision: str = "d0880aae6cee"
+down_revision: Union[str, None] = "2e5e90d3cdf8"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column("agents", sa.Column("compaction_settings", CompactionSettingsColumn(), nullable=True))
+
+
+def downgrade() -> None:
+    op.drop_column("agents", "compaction_settings")