Skip to content

test: surface pg_stat_statements in stress-test summary, fail on temp spills - #1341

Merged
porcellus merged 2 commits into
masterfrom
agent/issue-1337-stress-pgstat
Aug 2, 2026
Merged

test: surface pg_stat_statements in stress-test summary, fail on temp spills#1341
porcellus merged 2 commits into
masterfrom
agent/issue-1337-stress-pgstat

Conversation

@supertokens-agent-runner

Copy link
Copy Markdown
Contributor

Stacked on agent/issue-1336-stress-scale-paths (parent issue #1336). The diff
below is only my commit relative to that parent branch. Part of PLAN-006.

Problem

The stress-test Postgres already preloads pg_stat_statements with
track=all (stress-tests/docker-compose.yml), but nothing reads it — the
most useful diagnostic data of the run was discarded. Query-time and,
crucially, temp-file spill per statement were invisible.

Fix

Harvest pg_stat_statements at two points in the 1M-user run and turn it into
a signal that fails the run on spill regressions:

  • Two snapshots. capturePgStats('seed', { reset: true }) runs right after
    seeding completes (after createSessions) and captures the ingest profile,
    then resets the counters. capturePgStats('read') runs at the end of the run
    and captures the steady-state read/query profile. Resetting between them
    separates the two.
  • Per-phase tables rendered into both stats.json (new pgStatStatements
    section) and the workflow step summary: top ~15 statements by
    total_exec_time (with calls, mean time, rows), and every statement with
    temp_blks_written > 0 (with the spill volume). Query text is
    normalized-by-pg_stat_statements plus whitespace-collapsed, truncated to
    200 chars, and pipe/backslash-escaped for the Markdown cells. The harvester's
    own bookkeeping (… pg_stat_statements …) is filtered out.
  • Temp-spill tripwire. If any single read-phase statement writes more
    than STRESS_TEST_TEMP_BLKS_THRESHOLD blocks (default 10000 ≈ 80 MB), the
    run fails. Temp spill is hardware-independent, so this catches hash/sort-spill
    regressions deterministically even on noisy runners — where wall-clock
    budgets can't. It runs after stats.json and the summary are written, so
    the tables are visible even on a failing run (the workflow steps that render
    them are now if: always()).

All thresholds are env-overridable, mirroring the existing duration budgets:
STRESS_TEST_TEMP_BLKS_THRESHOLD (default 10000),
STRESS_TEST_ENFORCE_TEMP_SPILL=false (report-only, don't fail),
STRESS_TEST_PG_TOP_N (default 15),
STRESS_TEST_PG_CONNECTION_URI (default
postgresql://supertokens:supertokens@localhost:5432/supertokens — in CI the
compose publishes the db service's 5432 onto the runner's localhost). Adds
pg / @types/pg. CREATE EXTENSION IF NOT EXISTS pg_stat_statements is run
idempotently so the harvesting is self-contained.

Tests run locally

The full 1M-user suite is impractical to seed locally in-session, so I verified
the new, risky logic (SQL, reset semantics, rendering, threshold) end-to-end
against a real Postgres — a throwaway postgres:latest with
shared_preload_libraries=pg_stat_statements, work_mem=64kB — driving
capturePgStats directly (no 1M seed). On a clean DB:

  • npm run build (tsc) passes; prettier --check clean.
  • Seeding-phase snapshot captures ingest statements; after reset, the read
    phase's big spilling sort is absent from the seed snapshot — reset works
    within a run.
  • Read phase captures the temp spill (a 400k-row external sort → 9657
    temp_blks_written).
  • pg_stat_statements bookkeeping is filtered out of both phases.
  • Default threshold (10000) passes for the 9657-blk spill (acceptance:
    "defaults pass").
  • Threshold lowered to 1 fails the run with a clear error (acceptance:
    "an artificially low threshold demonstrably fails the run").
  • STRESS_TEST_ENFORCE_TEMP_SPILL=false reports without failing.
  • Rendered step-summary Markdown reviewed: both phases, both tables each,
    correct columns and formatting.

Not verified locally

  • A real 1M-user CI run (needs the seeded dataset + release image). The
    acceptance item "both phase tables visible in the step summary of a run" and
    "defaults pass on the current release image" should be confirmed by a
    workflow_dispatch run of the stress-tests workflow (enabled by parent
    Stress tests: re-enable assertions and allow manual dispatch #1335). The verification above exercises the same code path against real
    Postgres with a genuine spill, just without the 1M seed.
  • pg_stat_statements columns used (total_exec_time, mean_exec_time,
    temp_blks_written) are the PG 13+ names; the db service is
    postgres:latest, so this is fine, but it would need adjusting if the image
    were ever pinned to PG ≤ 12.

Cross-SDK note

Test-infra only (the stress-tests/ suite + its workflow); no library code and
no CDI/behavior change. Nothing to port to supertokens-node / -python / -golang.

Fixes #1337

… spills

Harvest the pg_stat_statements the stress-test Postgres already preloads but
nothing read. Snapshot the ingest profile when seeding finishes (then reset)
and the steady-state read/query profile at the end; render per-phase tables
(top statements by total execution time, and every statement that spilled to
temp) into stats.json and the workflow step summary; fail the run when a
read-phase statement writes more than an env-overridable number of temp blocks
(default 10k blocks ~= 80 MB). Temp spill is hardware-independent, so this
catches hash/sort-spill regressions deterministically even on noisy runners.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Base automatically changed from agent/issue-1336-stress-scale-paths to master August 2, 2026 06:51
@porcellus
porcellus marked this pull request as ready for review August 2, 2026 06:52
@porcellus
porcellus merged commit 866d51b into master Aug 2, 2026
9 checks passed
@porcellus
porcellus deleted the agent/issue-1337-stress-pgstat branch August 2, 2026 06:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stress tests: surface pg_stat_statements in the run summary and fail on temp spills

1 participant