Repository navigation
Commit 9665d04
Bench: frozen tool-only security bench for the review_checks.sh hub
bench/tool-only holds 11 fictional seeded files (CWE-89, 78, 798, 22, 327, 502, 489, 1104, 250, 94, 732), each with a clean twin, and an answer key committed before the scorer. scripts/bench_tool_only.py runs review_checks.sh --full over them in a throwaway repo and prints per-CWE recall, overall recall and false positives on the clean twins as JSON. Rows whose rule also fires on a clean twin do not count as hits. It fails below the frozen recall floor (0.909, 10/11 measured locally; the CWE-22 seed is matched only by a noise rule, a real hub gap) or above the frozen false-positive ceiling (27, measured in the CI real-hub job with the full tool set; a local run with fewer tools sees 8), and SKIPs when semgrep, gitleaks, trivy, bandit or ruff is absent, which fails under REQUIRE_REAL_HUB=1. The ci.yml real-hub job runs it after the tools install. The CWE-798 fixture is written by the scorer at run time from split parts, so no secret-shaped text is committed and no scanner allowlist is needed.
Tests: 3/3 (wired_check pass; bench_tool_only.py exit 0 locally, recall 0.909, 8 false positives; REQUIRE_REAL_HUB=1 without tools exits 1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>1 parent fc52e64 commit 9665d04
27 files changed
Lines changed: 201 additions & 0 deletions
File tree
- .github/workflows
- bench/tool-only
- clean
- .github/workflows
- vuln
- .github/workflows
- changelog.d
- scripts
- ci-tests.d
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
840 | 840 | | |
841 | 841 | | |
842 | 842 | | |
| 843 | + | |
| 844 | + | |
843 | 845 | | |
844 | 846 | | |
845 | 847 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
0 commit comments