Skip to content

feat(api-rs): retention for session_events - #1540

Open
0xAlcibiades wants to merge 3 commits into
paradigmxyz:mainfrom
0xAlcibiades:alcibiades/session-events-retention
Open

feat(api-rs): retention for session_events#1540
0xAlcibiades wants to merge 3 commits into
paradigmxyz:mainfrom
0xAlcibiades:alcibiades/session-events-retention

Conversation

@0xAlcibiades

Copy link
Copy Markdown
Contributor

Closes #1508

Takes the in-process sweep rather than pg_cron. Migration 0007's pg_cron
precedent is real, but retention that depends on an extension only the bundled
paradedb image ships would not work for deployments running their own Postgres,
and this is exactly the knob such a deployment needs.

Change

SESSION_EVENTS_RETENTION_DAYS (chart: apiRs.sessionEventsRetentionDays,
default 0 = off) deletes session_events past the window on the existing
cleanup sweep, so it inherits sandboxCleanupIntervalSecs rather than adding a
second schedule.

Off by default. session_events is durable history; starting to delete it
on an existing deployment because a version was bumped would be the wrong
default even though the growth is real.

Three properties worth reviewing

Events of a queued or running execution are never deleted, whatever their
age.
A long-running turn's early output is still needed to replay it, and age
alone does not separate "old" from "still in use". The left join also keeps
events whose execution row is gone (status is null), so an orphaned event is
retained rather than swept by accident.

Deletes are batched, and a sweep is bounded at 100k rows (5k × 20). The
first sweep after enabling retention can face millions of rows; one unbounded
delete would hold locks and bloat the table for its duration, and a sweep that
ran until done would hold the cleanup worker and its share of the connection
pool for as long. Successive sweeps drain the backlog.

Migration 0054 adds an index on created_at. The table had only
(thread_key, event_id) and (execution_id, event_type), so an age-keyed
delete would seq-scan the largest table in the schema on every pass — the
opposite of what a background sweep should do.

Note 0054 assumes 0053 is taken. If #1531 does not land first this should be
renumbered down.

Testing

Two unit tests: retention off by default without disabling the worker (the
sandbox arms still need it), and the per-sweep bound. The delete itself is
database-backed and unexercised here — I had no Postgres available, so the SQL
rests on review and CI rather than a local run. Flagging that rather than
implying otherwise.

cargo fmt --all --check and cargo clippy --all-targets -- -D warnings clean
across the three touched crates; cargo test passes. helm lint passes and
helm template renders the value.

Every event the control plane records lands in session_events, including one
row per harness stdout line, and nothing ever removed them: no TTL, no
partitioning, and no delete anywhere in api-rs. Rows only went via the
cascade from sessions, and sessions are not deleted in normal operation, so
the table grew monotonically for the life of the deployment -- order 1.1M
rows and 450MB a day on a modest single node, against a 20Gi default volume.

SESSION_EVENTS_RETENTION_DAYS deletes events past the window on the existing
cleanup sweep. Off by default: session_events is durable history and dropping
it is not something to start doing to an existing deployment unasked.

Events of a queued or running execution are never deleted, whatever their
age. A long-running turn's early output is still needed to replay it, and age
alone does not distinguish old from still-in-use.

Deletes are batched and each sweep is bounded at 100k rows. Switching
retention on for the first time can face millions of rows, and one unbounded
delete would hold locks and bloat the table for the duration; successive
sweeps drain the backlog instead.

Migration 0054 adds the created_at index the sweep needs -- the table had
only (thread_key, event_id) and (execution_id, event_type), so an age-keyed
delete would seq-scan the largest table in the schema on every pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

session_events grows without bound; no retention for per-line harness output

2 participants