test: two-size stress run with scaling-ratio assertions - #1342
Merged
Conversation
Run the measured read-path steps at two dataset sizes — a ~100k-user checkpoint mid-seed and the full 1M — and assert the per-step cost ratio time(1M)/time(100k) against an env-overridable per-class bound (O(1) ≤ 3×, O(n) ≤ 15×). This is hardware-independent: it catches superlinear scaling (the class this suite exists for) that absolute duration budgets can't distinguish from a slow runner. - Split the bulk import into a first ~100k checkpoint tranche and the remainder to 1M (importMillionUsers refactored into reusable helpers). - Extract the read-path steps into runReadPaths(deployment, size), run at both sizes via an ambient checkpoint context: the small pass records only into the ratio harness; the large pass feeds both the existing summary/budget table and the ratio harness. - RatioCollector computes clamped ratios (50ms floor), classifies each step, and fails the run when a ratio exceeds its bound; the globally destructive role-delete step runs only on the large pass and carries no ratio. - stats.json measurements are enriched with small/large/ratio; the workflow summary table gains 100k and ratio columns. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
porcellus
marked this pull request as ready for review
August 2, 2026 06:56
porcellus
approved these changes
Aug 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on
agent/issue-1337-stress-pgstat(issue #1337). The diff shows only this unit's commit relative to that parent; review against that base.Part of PLAN-006.
Problem
Absolute per-step duration budgets (added in the parent units) catch big regressions but can't tell a slow runner apart from pathological scaling — a step that is 3× slower on a slow CI box looks the same as one whose per-request cost is now growing with the dataset. The suite needs a hardware-independent check for the superlinear-growth class it exists to catch.
Approach
Run the measured read-path steps at two dataset sizes in a single environment (no second compose cycle) and assert the per-step cost ratio
time(1M)/time(100k)against a per-class bound:importMillionUsersis refactored into reusable helpers (createBulkImportRoles,listUserFiles,postBulkImportFiles,waitForBulkImport).index.tsimports the first ~100k checkpoint tranche (default 10 files × 10k, envSTRESS_TEST_SMALL_CHECKPOINT_FILES), runs the read paths, then imports the remainder to 1M and runs them again. The extra checkpoint is one short measurement pass, not a second seeding of 1M.index.ts) andmeasureQueryPathsare wrapped inrunReadPaths(deployment, size). An ambient checkpoint context routes eachmeasureTimecall: the small pass records only into the newRatioCollector(so the 100k numbers never pollute the 1M summary/budget table nor trip 1M budgets); the large pass feeds both the existingStatsCollector(unchanged summary/budget/pg_stat behaviour) and theRatioCollector.RatioCollectorclamps each measurement to a 50ms floor before dividing (so sub-noise can't manufacture a ratio), classifies each step O(1) (single-user lookups/writes, sign-in, first-page pagination, TOTP verify → bound 3×) or O(n) (counts, full pagination walk, analytics aggregates, large-share role listing → bound 15×), andthrowIfRatioExceeded()fails the run on any step over its bound. All bounds/floor/enforcement are env-overridable (STRESS_TEST_RATIO_O1_BOUND,STRESS_TEST_RATIO_ON_BOUND,STRESS_TEST_RATIO_BOUNDS,STRESS_TEST_RATIO_FLOOR_MS,STRESS_TEST_ENFORCE_RATIOS).stats.jsonmeasurements are enriched with small/large/ratio/bound/status, plus a top-levelscalingRatiosblock; the workflow summary table gains100kandRatio (1M/100k)columns.The one globally destructive step — deleting
role2, assigned to every bulk-imported user — runs only on the large pass and therefore carries no ratio; every other read-path step is read-only or creates its own fresh fixtures, so it repeats cleanly across passes.Tests / verification run locally
throwIfRatioExceeded()throwing when a step is over bound,STRESS_TEST_ENFORCE_RATIOS=falsereporting without throwing, and thewriteToFilemerge attaching small/ratio columns to the matching measurement. All checks passed.supertokens/supertokens-postgresql:latestvia the suite'sdocker-compose.yml, at tiny scale (two 40-user tranches,STRESS_TEST_TOTP_USED_CODES=5): the two-tranche import + two-pass flow ran green, produced 24 two-size ratio rows instats.jsonwith small/large/ratio and ✅ markers, correctly leftDelete role (large share)with no ratio, and emitted thescalingRatiosblock.tsc --noEmitand the suite'sprettier --checkboth pass.Not verified locally
The full 1M two-size run is CI-only (needs a published dev image and ~an hour). Reviewers can trigger it via the
workflow_dispatchthe hygiene unit added. To reproduce the "fails when a bound is set artificially low" acceptance on real data, dispatch with e.g.STRESS_TEST_RATIO_O1_BOUND=1. The default bounds (3×/15×) are placeholders that, like the parent units' duration budgets and temp-spill threshold, should be calibrated from a recorded baseline run (PLAN-006 rollout checklist).Cross-SDK note
Test-infrastructure only, specific to supertokens-core's
stress-tests/suite — no product code and nothing to port to supertokens-node/python/golang.Fixes #1338