Skip to content

test: two-size stress run with scaling-ratio assertions - #1342

Merged
porcellus merged 2 commits into
masterfrom
agent/issue-1338-stress-ratio
Aug 2, 2026
Merged

test: two-size stress run with scaling-ratio assertions#1342
porcellus merged 2 commits into
masterfrom
agent/issue-1338-stress-ratio

Conversation

@supertokens-agent-runner

Copy link
Copy Markdown
Contributor

Stacked on agent/issue-1337-stress-pgstat (issue #1337). The diff shows only this unit's commit relative to that parent; review against that base.

Part of PLAN-006.

Problem

Absolute per-step duration budgets (added in the parent units) catch big regressions but can't tell a slow runner apart from pathological scaling — a step that is 3× slower on a slow CI box looks the same as one whose per-request cost is now growing with the dataset. The suite needs a hardware-independent check for the superlinear-growth class it exists to catch.

Approach

Run the measured read-path steps at two dataset sizes in a single environment (no second compose cycle) and assert the per-step cost ratio time(1M)/time(100k) against a per-class bound:

  • Import in two tranches. importMillionUsers is refactored into reusable helpers (createBulkImportRoles, listUserFiles, postBulkImportFiles, waitForBulkImport). index.ts imports the first ~100k checkpoint tranche (default 10 files × 10k, env STRESS_TEST_SMALL_CHECKPOINT_FILES), runs the read paths, then imports the remainder to 1M and runs them again. The extra checkpoint is one short measurement pass, not a second seeding of 1M.
  • One read-path definition, run twice. The pagination/count steps (were inline in index.ts) and measureQueryPaths are wrapped in runReadPaths(deployment, size). An ambient checkpoint context routes each measureTime call: the small pass records only into the new RatioCollector (so the 100k numbers never pollute the 1M summary/budget table nor trip 1M budgets); the large pass feeds both the existing StatsCollector (unchanged summary/budget/pg_stat behaviour) and the RatioCollector.
  • Ratio + gate. RatioCollector clamps each measurement to a 50ms floor before dividing (so sub-noise can't manufacture a ratio), classifies each step O(1) (single-user lookups/writes, sign-in, first-page pagination, TOTP verify → bound 3×) or O(n) (counts, full pagination walk, analytics aggregates, large-share role listing → bound 15×), and throwIfRatioExceeded() fails the run on any step over its bound. All bounds/floor/enforcement are env-overridable (STRESS_TEST_RATIO_O1_BOUND, STRESS_TEST_RATIO_ON_BOUND, STRESS_TEST_RATIO_BOUNDS, STRESS_TEST_RATIO_FLOOR_MS, STRESS_TEST_ENFORCE_RATIOS).
  • Reporting. stats.json measurements are enriched with small/large/ratio/bound/status, plus a top-level scalingRatios block; the workflow summary table gains 100k and Ratio (1M/100k) columns.

The one globally destructive step — deleting role2, assigned to every bulk-imported user — runs only on the large pass and therefore carries no ratio; every other read-path step is read-only or creates its own fresh fixtures, so it repeats cleanly across passes.

Tests / verification run locally

  • Ratio-logic unit harness (throwaway, not committed): verified clamped-ratio math, per-class default bounds (3 / 15), tiny-measurement floor clamp, large-only steps producing no ratio row, throwIfRatioExceeded() throwing when a step is over bound, STRESS_TEST_ENFORCE_RATIOS=false reporting without throwing, and the writeToFile merge attaching small/ratio columns to the matching measurement. All checks passed.
  • Live end-to-end smoke (throwaway, not committed) against supertokens/supertokens-postgresql:latest via the suite's docker-compose.yml, at tiny scale (two 40-user tranches, STRESS_TEST_TOTP_USED_CODES=5): the two-tranche import + two-pass flow ran green, produced 24 two-size ratio rows in stats.json with small/large/ratio and ✅ markers, correctly left Delete role (large share) with no ratio, and emitted the scalingRatios block. tsc --noEmit and the suite's prettier --check both pass.

Not verified locally

The full 1M two-size run is CI-only (needs a published dev image and ~an hour). Reviewers can trigger it via the workflow_dispatch the hygiene unit added. To reproduce the "fails when a bound is set artificially low" acceptance on real data, dispatch with e.g. STRESS_TEST_RATIO_O1_BOUND=1. The default bounds (3×/15×) are placeholders that, like the parent units' duration budgets and temp-spill threshold, should be calibrated from a recorded baseline run (PLAN-006 rollout checklist).

Cross-SDK note

Test-infrastructure only, specific to supertokens-core's stress-tests/ suite — no product code and nothing to port to supertokens-node/python/golang.

Fixes #1338

Run the measured read-path steps at two dataset sizes — a ~100k-user
checkpoint mid-seed and the full 1M — and assert the per-step cost ratio
time(1M)/time(100k) against an env-overridable per-class bound (O(1) ≤ 3×,
O(n) ≤ 15×). This is hardware-independent: it catches superlinear scaling
(the class this suite exists for) that absolute duration budgets can't
distinguish from a slow runner.

- Split the bulk import into a first ~100k checkpoint tranche and the
  remainder to 1M (importMillionUsers refactored into reusable helpers).
- Extract the read-path steps into runReadPaths(deployment, size), run at
  both sizes via an ambient checkpoint context: the small pass records only
  into the ratio harness; the large pass feeds both the existing
  summary/budget table and the ratio harness.
- RatioCollector computes clamped ratios (50ms floor), classifies each step,
  and fails the run when a ratio exceeds its bound; the globally destructive
  role-delete step runs only on the large pass and carries no ratio.
- stats.json measurements are enriched with small/large/ratio; the workflow
  summary table gains 100k and ratio columns.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Base automatically changed from agent/issue-1337-stress-pgstat to master August 2, 2026 06:52
@porcellus
porcellus marked this pull request as ready for review August 2, 2026 06:56
@porcellus
porcellus merged commit 04e5dfa into master Aug 2, 2026
9 checks passed
@porcellus
porcellus deleted the agent/issue-1338-stress-ratio branch August 2, 2026 06:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stress tests: two-size run with scaling-ratio assertions

1 participant