Skip to content

session_events grows without bound; no retention for per-line harness output #1508

Description

@0xAlcibiades

What happens

Every event the control plane records for a session is persisted in session_events, including one row per harness stdout line, and there is no retention mechanism of any kind: no TTL, no partitioning, and no delete from session_events anywhere in api-rs. Rows only go away via the on delete cascade from sessions, and sessions are not deleted in normal operation, so the table grows monotonically for the life of the deployment.

Why it hurts

On a modest single-node deployment we observed ~1.1M session_events rows per day, roughly 450MB+/day of Postgres growth — against the chart's default 20Gi Postgres PVC that is weeks to a full disk on the control-plane database. The bulk of the volume is session.output.line rows (per-delta harness output), which are read back for live streaming and recovery of recent executions; multi-week-old output lines for long-terminal executions are dead weight.

Where in the code

Worth noting for implementation: the bundled database is paradedb/paradedb with shared_preload_libraries=pg_search,pg_cron (contrib/chart/values.yaml#L730-L739), and migration 0007_absurd_workflows.sql already builds pg_cron-based partition maintenance for the workflow tables — so there is in-tree precedent for scheduled in-database cleanup.

Proposed direction

A configurable retention window for session_events, default off for compatibility: delete events older than N days that belong to terminal executions (or simply all events past the window). Either an in-process sweep on the existing cleanup interval or an opt-in pg_cron job following the migration-0007 pattern would fit; whichever lands should be documented in the configuration reference so operators size the Postgres PVC deliberately instead of discovering the growth rate from a full disk.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions