Repository navigation
Add tool-only bench scorer and real-hub CI step - #2382
Merged
Merged
Conversation
Eleven small fictional seeded files across CWE classes (SQLi, command injection, hardcoded secret, path traversal, weak crypto, insecure deserialization, debug on, vulnerable pinned dependency, Dockerfile root, GitHub Actions injection, Terraform public bucket) with a clean twin per class and answer.json mapping file:line to CWE. Fixtures only; the scorer follows. Tests: 0/0 (data only) No-Mechanism-Reason: fixtures and answer key only, no skill prose; the scorer lands in the next commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/bench_tool_only.py runs review_checks.sh --full over the frozen bench/tool-only fixtures and prints per-CWE recall, overall recall and false positives on the clean twins as JSON. It fails below the frozen recall floor (1.0, 11/11 measured locally) or above the frozen false-positive ceiling (8), and exits 0 with SKIP when semgrep, gitleaks or trivy is absent. The ci.yml real-hub job runs it after the tools install. Floors are local measurements; confirm them against the first CI run. Tests: 1/1 (bench_tool_only.py exit 0 locally, recall 1.0, 8 false positives) No-Mechanism-Reason: no skill SKILL.md or references edited; this commit adds a bench script and CI step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tests: 2/2 No-Mechanism-Reason: config allowlist for a fictional fixture; no skill prose edited. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SKIP exits 1 under REQUIRE_REAL_HUB=1 (ci.yml step now sets it). Removed BENCH_FLOOR_RECALL/BENCH_MAX_FP overrides. Rows whose rule also fires on the clean twins no longer count as hits; line-0 rows count only for the dependency fixture. bandit and ruff added to CORE (the real-hub job installs both). Recall re-frozen to 10/11 (0.909): CWE-22 was matched only by a noise rule. Tests: bench_tool_only.py local run exit 0; REQUIRE_REAL_HUB=1 minimal PATH exit 1 (SKIP) No-Mechanism-Reason: bench script only; no skill prose edited. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tests: 2/2 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move vuln/secret.py out of the tree: bench_tool_only.py writes it into the temp dir from split literals on line 2, so no committed text matches a secret pattern. Drops the gitleaks allowlist entry; answer.json keys unchanged. Tests: 3/3 No-Mechanism-Reason: bench fixture move; no skill prose edited. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tests: 2/2 Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Tests: 1/1 (python3 -m py_compile scripts/bench_tool_only.py) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
scripts/bench_tool_only.py runs review_checks.sh --full over the frozen bench/tool-only fixtures and prints per-CWE recall, overall recall and false positives on the clean twins as JSON. It fails below the frozen recall floor (1.0, 11/11 measured locally) or above the frozen false-positive ceiling (8), and exits 0 with SKIP when semgrep, gitleaks or trivy is absent. The ci.yml real-hub job runs it after the tools install. Floors are local measurements; confirm them against the first CI run.
Tests: 1/1 (bench_tool_only.py exit 0 locally, recall 1.0, 8 false positives)
No-Mechanism-Reason: no skill SKILL.md or references edited; this commit adds a bench script and CI step.
🤖 Generated with Claude Code