Explore REST API fuzzing with Schemathesis

[strickvl]

Explore Schemathesis against Kitaru's OpenAPI-defined REST API as a bounded local experiment, separate from the coverage-guided fuzzing work in #907 and the non-REST properties in #910.

Try generated valid and invalid requests against isolated application state, record reproducible failures, and distinguish server defects from schema gaps and test-harness limitations. Use the results to decide which endpoints and checks warrant a small repeatable test suite, whether PostgreSQL or in-memory fixtures are needed, and whether stateful create/read/update/delete sequences add useful coverage.

This is an exploratory task, not a commitment to CI integration. Keep runs local with disposable data, no cloud services or model calls, and no execution of generated plugin code. Report security-sensitive findings privately.

[strickvl]

Spike results

Ran a bounded Schemathesis 4.25.2 spike against the real app. Harness is on the spike/api-fuzzing branch under tests/fuzz_spike/ — not wired into CI, gated behind KITARU_FUZZ=1, and schemathesis is deliberately not in pyproject.toml yet.

It works with no changes to the server. All 141 operations ran, nothing hung, nothing needed an external service or a model call. The positive pass takes 39 seconds, which makes deeper runs affordable.

Setup: real create_app(), real Alembic migrations, a disposable PostgreSQL database per run on localhost:5433, and a real bearer token from POST /api/v1/login as the bootstrapped default admin — no dependency override, so foreign-key-constrained writes actually work. Analytics, S3, control plane, and OpenTelemetry all quiet; task sweeper off.

One design note worth reviewing before this becomes a keepable suite: the app runs under uvicorn in a background thread rather than through Schemathesis's in-process ASGI transport. That transport enters and exits its client context around every request, which re-runs the whole lifespan — Alembic migrations included — per generated example. One uvicorn server per pytest session runs the lifespan once.

Four passes

Pass	Examples/op	Checks	Result
Positive	25	not_a_server_error, response_schema_conformance	47 failed / 94 passed, 43s
Negative	25	not_a_server_error, negative_data_rejection	2 failed / 134 passed / 5 skipped, 167s
500-hunt	50, both modes	not_a_server_error only	6 failed / 135 passed, 258s
Stateful (agents)	30, 8 steps	not_a_server_error, use_after_free	passes

The 47 positive failures dedupe to 3 root causes. Every finding below was independently re-reproduced with a plain httpx call against a fresh server before being filed.

Filed

• NUL byte in any user string reaches PostgreSQL and 500s (device_authorization unauthenticated, imports) #927 — NUL byte in any user string reaches PostgreSQL and 500s. A class, not two endpoints; the device_authorization instance needs no authentication.
• Out-of-range version path parameter 500s instead of 404 on evaluator and importer version routes #928 — out-of-range version path parameter overflows the INTEGER column and 500s on four routes.
• POST /api/v1/login validation errors omit the required error field from their declared 400 body #929 — POST /api/v1/login validation errors omit the error field their own declared 400 body requires.
• OpenAPI declares only the array-shaped 422 body, but hand-raised validation errors return a string detail #930 — the spec declares only the array-shaped 422 body while hand-raised validation errors return a string detail. One signature, 43 of the 47 positive failures.

One negative-pass failure was a false alarm and is not filed: POST /api/v1/device_authorization answering 200 to an "invalid" form body is correct, because every field there is optional and urlencoding makes the generated body schema-valid by the time it arrives.

Limits of this run, for whoever scopes the real suite

Depth was shallow. 25 examples per operation, 50 for the crash hunt. The NUL-byte bug appeared only at 50 — it was not found at 25. Direct evidence that more of the same class is waiting. A 500-example run is roughly 20-30 minutes: nightly territory, not a PR gate.

No authorization testing is possible against this spec at all. Every request ran as an admin. More fundamentally, components.securitySchemes is empty and no operation declares security:, so the spec never tells any tool that authentication exists. Schemathesis's ignored_auth check is therefore inert until that is fixed. The worker-only and task-only dependency variants were never exercised.

status_code_conformance is unusable as-is. The complete set of declared status codes across the whole spec is {200, 201, 204, 400, 422}, so every legitimate 404, 401, 403, 409, and 503 reads as a violation. Excluded rather than triaged; folded into #930.

The filter query parameter is effectively untested. It is a recursive JSON-encoded filter language in a query string, present on every list endpoint, and the highest-signal parser target in the API. Schema-driven generation found nothing there, which should not be read as evidence it is sound — a language parser wants a grammar-aware fuzzer.

Free-form body endpoints are barely scratched. POST/PATCH /api/v1/sessions (inputs/outputs completely untyped), POST /api/v1/session-runs, POST /api/v1/sessions/{id}/nodes, annotations value, tasks result, imports params. Because their schemas are untyped object, Hypothesis generates mostly trivial values. They want hand-written strategies.

Stateful coverage is thin. One resource pair (agents plus agent-versions), and the spec declares no OpenAPI links, so chains come purely from inference off Location headers and matching id fields. "It passed" means "no 500 on shallow inferred create/read/delete chains for one resource", not a meaningful stateful result.

POST /api/v1/workers got zero negative coverage — Hypothesis raised Unsatisfiable in negative mode, presumably a constraint in WorkerCreateRequest the generator cannot invert. Not chased down.

No security-oriented passes: no SQL-injection payloads through sort/filter, no path traversal, no oversized bodies against POST /api/v1/blobs, no header injection. Out of scope for a schema-driven spike, but worth naming.

Two settings that matter for anything adopted from this: keep Schemathesis's allow_x00=True, since turning it off hides the entire #927 family, and note that the whole family is invisible on SQLite — these tests need PostgreSQL. All four passes ran with derandomize=True for reproducibility, so repeated runs explore the same inputs.

[strickvl]

Deep run: depth pays

Reran the crash hunt at 400 examples per operation in both generation modes, randomized rather than derandomized so it explores inputs the earlier passes never saw. 11 minutes and 40 seconds for all 141 operations.

10 failed / 131 passed, against 6 failures at 50 examples per operation.

The four new failures were not noise:

• Reusing an Idempotency-Key across routes 500s on POST /api/v1/api-keys (decrypt runs before the fingerprint check) #931 — a genuinely new bug class. Reusing one Idempotency-Key across routes crashes POST /api/v1/api-keys with a DecryptionError, because the stored row is decrypted before the fingerprint check that would have rejected it. Nothing about this resembles the NUL-byte or integer-overflow families; it would not have been found by hardening either of those.
• Two more instances of NUL byte in any user string reaches PostgreSQL and 500s (device_authorization unauthenticated, imports) #927, including a second unauthenticated one on POST /api/v1/login, and one raising a different asyncpg exception type (UntranslatableCharacterError from a JSON column rather than CharacterNotInRepertoireError from a text column) — which the proposed error-handler backstop has to catch too, or that instance still 500s after the fix.
• The two PATCH siblings of Out-of-range version path parameter 500s instead of 404 on evaluator and importer version routes #928, which the shallower run reached only on their GET counterparts.

What this says about how to run it

The finding-versus-depth curve is the useful result here, and it is steeper than expected:

Examples/op	Distinct failures	Wall clock
25	3 root causes	43s
50	6	258s
400 (randomized)	10	700s

Every step up found something the step below missed, and the newest finding is a class the others do not cover. That is the case for nightly depth over a shallow gate: at 25 examples per operation this suite would have found neither #927 nor #931.

Two settings mattered. Turning off derandomize is what let this run explore fresh inputs rather than replay the same ones — a derandomized nightly would re-prove yesterday's findings forever. And the existing fuzz-nightly.yml already caches the Hypothesis database between runs, which is the right complement: randomized exploration finds new inputs, the cached database replays known failures first so a regression is caught immediately.

Harness changes made while running this

Two small ones on spike/api-fuzzing, both worth keeping if this suite is adopted:

• KITARU_FUZZ_RANDOM=1 turns off derandomize for exploratory runs; the default stays derandomized so a gate is reproducible.
• The exception-capture file path is now KITARU_FUZZ_CAPTURE rather than a hardcoded absolute path.

Reproduce this run with:

KITARU_FUZZ=1 KITARU_FUZZ_RANDOM=1 KITARU_FUZZ_HUNT_EXAMPLES=400 \
  uv run pytest tests/fuzz_spike/test_schemathesis_fuzz.py -k hunt -p no:randomly