test: surface pg_stat_statements in stress-test summary, fail on temp spills - #1341
Merged
Conversation
… spills Harvest the pg_stat_statements the stress-test Postgres already preloads but nothing read. Snapshot the ingest profile when seeding finishes (then reset) and the steady-state read/query profile at the end; render per-phase tables (top statements by total execution time, and every statement that spilled to temp) into stats.json and the workflow step summary; fail the run when a read-phase statement writes more than an env-overridable number of temp blocks (default 10k blocks ~= 80 MB). Temp spill is hardware-independent, so this catches hash/sort-spill regressions deterministically even on noisy runners. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
porcellus
approved these changes
Aug 2, 2026
porcellus
marked this pull request as ready for review
August 2, 2026 06:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on
agent/issue-1336-stress-scale-paths(parent issue #1336). The diffbelow is only my commit relative to that parent branch. Part of PLAN-006.
Problem
The stress-test Postgres already preloads
pg_stat_statementswithtrack=all(stress-tests/docker-compose.yml), but nothing reads it — themost useful diagnostic data of the run was discarded. Query-time and,
crucially, temp-file spill per statement were invisible.
Fix
Harvest
pg_stat_statementsat two points in the 1M-user run and turn it intoa signal that fails the run on spill regressions:
capturePgStats('seed', { reset: true })runs right afterseeding completes (after
createSessions) and captures the ingest profile,then resets the counters.
capturePgStats('read')runs at the end of the runand captures the steady-state read/query profile. Resetting between them
separates the two.
stats.json(newpgStatStatementssection) and the workflow step summary: top ~15 statements by
total_exec_time(with calls, mean time, rows), and every statement withtemp_blks_written > 0(with the spill volume). Query text isnormalized-by-
pg_stat_statementsplus whitespace-collapsed, truncated to200 chars, and pipe/backslash-escaped for the Markdown cells. The harvester's
own bookkeeping (
… pg_stat_statements …) is filtered out.than
STRESS_TEST_TEMP_BLKS_THRESHOLDblocks (default10000≈ 80 MB), therun fails. Temp spill is hardware-independent, so this catches hash/sort-spill
regressions deterministically even on noisy runners — where wall-clock
budgets can't. It runs after
stats.jsonand the summary are written, sothe tables are visible even on a failing run (the workflow steps that render
them are now
if: always()).All thresholds are env-overridable, mirroring the existing duration budgets:
STRESS_TEST_TEMP_BLKS_THRESHOLD(default 10000),STRESS_TEST_ENFORCE_TEMP_SPILL=false(report-only, don't fail),STRESS_TEST_PG_TOP_N(default 15),STRESS_TEST_PG_CONNECTION_URI(defaultpostgresql://supertokens:supertokens@localhost:5432/supertokens— in CI thecompose publishes the
dbservice's 5432 onto the runner's localhost). Addspg/@types/pg.CREATE EXTENSION IF NOT EXISTS pg_stat_statementsis runidempotently so the harvesting is self-contained.
Tests run locally
The full 1M-user suite is impractical to seed locally in-session, so I verified
the new, risky logic (SQL, reset semantics, rendering, threshold) end-to-end
against a real Postgres — a throwaway
postgres:latestwithshared_preload_libraries=pg_stat_statements,work_mem=64kB— drivingcapturePgStatsdirectly (no 1M seed). On a clean DB:npm run build(tsc) passes;prettier --checkclean.phase's big spilling sort is absent from the seed snapshot — reset works
within a run.
9657temp_blks_written).pg_stat_statementsbookkeeping is filtered out of both phases."defaults pass").
1fails the run with a clear error (acceptance:"an artificially low threshold demonstrably fails the run").
STRESS_TEST_ENFORCE_TEMP_SPILL=falsereports without failing.correct columns and formatting.
Not verified locally
acceptance item "both phase tables visible in the step summary of a run" and
"defaults pass on the current release image" should be confirmed by a
workflow_dispatchrun of the stress-tests workflow (enabled by parentStress tests: re-enable assertions and allow manual dispatch #1335). The verification above exercises the same code path against real
Postgres with a genuine spill, just without the 1M seed.
pg_stat_statementscolumns used (total_exec_time,mean_exec_time,temp_blks_written) are the PG 13+ names; thedbservice ispostgres:latest, so this is fine, but it would need adjusting if the imagewere ever pinned to PG ≤ 12.
Cross-SDK note
Test-infra only (the
stress-tests/suite + its workflow); no library code andno CDI/behavior change. Nothing to port to supertokens-node / -python / -golang.
Fixes #1337