Add grammar-aware filter fuzzing - #958
Open
strickvl wants to merge 13 commits into
Open
Conversation
Exploratory harness for #923: runs the real app (lifespan, Alembic migrations, disposable Postgres database, real bearer-token login) under uvicorn in a background thread and fuzzes all 141 operations from openapi/openapi.json with Schemathesis 4.x. Spike-only code, not wired into CI. Requires 'uv pip install schemathesis' (deliberately not added to pyproject.toml yet) and 'docker compose up -d db'.
Deep exploratory runs want fresh inputs, so KITARU_FUZZ_RANDOM=1 turns off derandomize; the default stays derandomized for reproducibility. The exception capture path was hardcoded to an absolute scratch path.
Generates requests from openapi/openapi.json in both schema-conformant and schema-violating modes and sends them to the real app on a disposable PostgreSQL database, asserting only that the server never answers 5xx. One session-scoped database is shared across the run, which makes "was this rejected?" depend on run order while leaving "did it crash?" well-posed, so negative_data_rejection is not run. Response schema conformance is gated behind an env var until #930 lands. The four defects this found are listed in KNOWN_FAILURES and skip, so a filed crash does not mask the rest of its operation.
# Conflicts: # uv.lock
#930 fixed the 422 response schemas, so the KITARU_FUZZ_SCHEMA_CONFORMANCE gate on test_response_matches_schema no longer has a reason to exist. The full suite now passes with the check on: 282 passed, 0 skipped.
tests/server/test_fuzz_api.py imports schemathesis, which lives in the fuzz dependency group. Without it, ty cannot resolve the import and the job fails with four unresolved-import diagnostics.
The module-level skipif only applies after the module imports, and the regular test matrix does not install the fuzz group, so pytest crashed during collection with ModuleNotFoundError. importorskip before the schemathesis imports skips the whole module instead, matching test_otel.
The nightly workflow caches .hypothesis/examples and uploads it as the replay artifact named in the auto-filed issue. database=None left that artifact empty for the api surface, so a randomized failure could not be replayed as the issue instructs.
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Recursive list filters now get their own nightly property run. It exercises filter JSON through the Agent list boundary and PostgreSQL SQL compilation, so the recursive grammar cannot silently drift outside the schema-based API fuzzer's coverage.
Related: #907
Related: #932
Reviewer Notes
The valid cases are deliberately limited to the API's accepted depth and condition caps. They cover every boolean node and Agent filter operator without a live database. A separate hostile-shape property permits normal Pydantic validation failures but rejects crashes such as
RecursionError.Reproduction
The nightly profile ran 2,000 examples for each property in 4.16 seconds.
just checkalso passed.