Valid sessions intermittently receive 401s under load

[spendres]

Preflight checklist

• I could not find a solution in the existing issues, docs, nor discussions.
• I agree to follow this project's Code of Conduct.
• I have read and am following this repository's Contribution Guidelines.
• I have joined the Ory Community Slack.
• I am signed up to the Ory Security Patch Newsletter.

Ory Network Project

No response

Describe the bug

Under sustained high-throughput session validation (~2,700 GET /sessions/whoami req/s against one Kratos instance), a small fraction of
requests (~2–4%) intermittently receive 401 with:

{"error":{"code":401,"status":"Unauthorized","reason":"No valid session credentials found in the request.","message":"The request could not be authorized"}}

for session tokens that are provably valid:

• The same tokens validated successfully (200, active: true) minutes
  earlier in the same process lifetime.
• Replaying the exact rejected token 100ms later returns 200 for ~63% of
  rejections (measured across 1,539 rejections in one run); replaying minutes
  later returns 200 with active: true and an expires_at ~30 days out.
• Kratos's own access log for the failing requests shows the
  X-Session-Token header present and well-formed, the 401 served in ~11ms,
  ory-error-id empty, and no level=error/level=warning entries
  correlated with the window.

So the same token flaps invalid → valid within 100ms. The flap is per-lookup,
not per-session-state.

Reproducing the bug

We can reproduce it reliably inside a large load test but have NOT found a
minimal reproduction — including two isolation attempts that were 100% clean
(details below), which is why we're reporting the full shape.

Reproducing context (happens every run of this shape, 5+ reproductions):

1. Provision 200k–1M fresh identities against a clean Kratos+Postgres
   (either password registration + login, or Admin-API identity creation +
   passwordless code login — the failure occurs identically with both, so
   it is not credential-type specific). Each identity holds 1–2 sessions;
   sessions table reaches ~0.4–2M rows.
2. Shortly after provisioning completes (~2–8 minutes), begin a sustained
   burst of authenticated API calls from a single backend service: each call
   triggers one GET /sessions/whoami (native, X-Session-Token), iterating
   through the population's distinct tokens at ~2,700 req/s.
3. ~55–60 seconds into the burst (strikingly consistently around the
   150k–170k-th request), a window of ~6–20+ seconds opens in which ~2–4% of
   whoami calls return the 401 above. Client retries with backoff (5 × 2s)
   ride it out; the run then completes normally.

Isolation attempts that did NOT reproduce (same instance, same data):

• 857,586 whoami requests at ~9,500 req/s with a single valid token →
  100% HTTP 200.
• 815,178 whoami requests at ~9,000 req/s rotating 5,992 distinct
  freshly-minted tokens → 100% HTTP 200.

So neither raw rate (3.5× higher than the failing storm) nor token diversity
alone triggers it. The full context — a large just-provisioned population
plus a sustained many-distinct-token storm — appears necessary.

Ruled out on our side:

• Client/header loss: Kratos's access log shows the token on the failing
  requests; no client-side exceptions (clean 401 JSON responses).
• Session expiry/revocation: lifespan: 720h; the same tokens work
  before, during (on retry), and after the window; nothing revokes sessions.
• Postgres long transactions / stale snapshots: pg_stat_activity sampled
  every 8s during a failing run — zero idle-in-transaction backends, max
  transaction age 9.2s.
• DB overload: the 401s are served in ~11ms (fast, clean "not found"-shaped
  responses, not timeouts); Postgres CPU was unremarkable; a 500 would be
  expected for persister errors, and we see none.

Relevant log output

Failing request as logged by Kratos (debug level; identifying values
redacted/shortened):


time=<T> level=info msg=completed handling request http_request=map[headers:map[accept:application/json ... x-session-token:[ory_st_Md5gmr…]] host:localhost:4433 method:GET path:/sessions/whoami ...] http_response=map[headers:map[... ory-error-id:] size:157 status:401 text_status:Unauthorized took:11.364333ms]


Same token replayed later:


{"id":"653e5bcf-…","active":true,"expires_at":"2026-08-02T01:11:07Z","authenticated_at":"2026-07-03T01:11:07Z","authenticator_assurance_level":"aal1", ...}

Relevant configuration

dsn: postgres://kratos:<SECRET>@<SECRET>:<PORT>/kratos?sslmode=disable&max_conns=20&max_idle_conns=4
session:
  lifespan: 720h
  earliest_possible_extend: 24h
selfservice:
  methods:
    passkey: { enabled: true }
    password: { enabled: true }
    code: { enabled: true, passwordless_enabled: true }
    totp: { enabled: true }
hashers:
  algorithm: bcrypt
  bcrypt: { cost: 8 }
feature_flags:
  use_continue_with_transitions: true

Version

v26.2.0 (docker oryd/kratos:v26.2.0)

On which operating system are you observing this issue?

macOS

In which environment are you deploying?

Docker

Additional Context

macOS host (Apple Silicon), Kratos in Docker Desktop's linux/arm64 VM.
Docker VM: 10 CPUs / 44GB. Postgres 16.13 in the same compose network.

• The whoami traffic originates from one backend process (Deno) using
  fetch with keep-alive to http://localhost:4433, i.e. the VM's
  host-gateway path. The two clean isolation hammers used the same client
  stack and network path.
• The onset consistency is the strangest datum: across three runs at
  different population sizes (200k, 200k, 500k), the window opened at
  request ~152k, ~156k, ~164k, ~168k of the storm — suspiciously near
  163,840 = 5×2^15, suggesting some capacity/structure boundary rather than
  timing.
• Happy to run instrumented builds, enable specific tracing, or attempt
  reproductions with adjusted configs (e.g. different max_conns, no
  --dev, no --watch-courier) if that helps narrow it down. The full
  investigation timeline can be shared if helpful.