Skip to content

Add tool-only bench scorer and real-hub CI step - #2382

Merged
remigiusz-antczak merged 9 commits into
mainfrom
feat/hub-real-smoke-18
Nov 15, 2026
Merged

remigiusz-antczak merged 9 commits into
mainfrom
feat/hub-real-smoke-18

Conversation

@remigiusz-antczak

Copy link
Copy Markdown
Collaborator

scripts/bench_tool_only.py runs review_checks.sh --full over the frozen bench/tool-only fixtures and prints per-CWE recall, overall recall and false positives on the clean twins as JSON. It fails below the frozen recall floor (1.0, 11/11 measured locally) or above the frozen false-positive ceiling (8), and exits 0 with SKIP when semgrep, gitleaks or trivy is absent. The ci.yml real-hub job runs it after the tools install. Floors are local measurements; confirm them against the first CI run.

Tests: 1/1 (bench_tool_only.py exit 0 locally, recall 1.0, 8 false positives)
No-Mechanism-Reason: no skill SKILL.md or references edited; this commit adds a bench script and CI step.

🤖 Generated with Claude Code

remigiusz-antczak and others added 9 commits November 15, 2026 14:44
Eleven small fictional seeded files across CWE classes (SQLi, command injection, hardcoded secret, path traversal, weak crypto, insecure deserialization, debug on, vulnerable pinned dependency, Dockerfile root, GitHub Actions injection, Terraform public bucket) with a clean twin per class and answer.json mapping file:line to CWE. Fixtures only; the scorer follows.

Tests: 0/0 (data only)
No-Mechanism-Reason: fixtures and answer key only, no skill prose; the scorer lands in the next commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/bench_tool_only.py runs review_checks.sh --full over the frozen bench/tool-only fixtures and prints per-CWE recall, overall recall and false positives on the clean twins as JSON. It fails below the frozen recall floor (1.0, 11/11 measured locally) or above the frozen false-positive ceiling (8), and exits 0 with SKIP when semgrep, gitleaks or trivy is absent. The ci.yml real-hub job runs it after the tools install. Floors are local measurements; confirm them against the first CI run.

Tests: 1/1 (bench_tool_only.py exit 0 locally, recall 1.0, 8 false positives)
No-Mechanism-Reason: no skill SKILL.md or references edited; this commit adds a bench script and CI step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tests: 2/2
No-Mechanism-Reason: config allowlist for a fictional fixture; no skill prose edited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SKIP exits 1 under REQUIRE_REAL_HUB=1 (ci.yml step now sets it). Removed
BENCH_FLOOR_RECALL/BENCH_MAX_FP overrides. Rows whose rule also fires on the
clean twins no longer count as hits; line-0 rows count only for the dependency
fixture. bandit and ruff added to CORE (the real-hub job installs both).
Recall re-frozen to 10/11 (0.909): CWE-22 was matched only by a noise rule.

Tests: bench_tool_only.py local run exit 0; REQUIRE_REAL_HUB=1 minimal PATH exit 1 (SKIP)
No-Mechanism-Reason: bench script only; no skill prose edited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tests: 2/2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move vuln/secret.py out of the tree: bench_tool_only.py writes it into the temp dir from split literals on line 2, so no committed text matches a secret pattern. Drops the gitleaks allowlist entry; answer.json keys unchanged.

Tests: 3/3
No-Mechanism-Reason: bench fixture move; no skill prose edited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tests: 2/2

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Tests: 1/1 (python3 -m py_compile scripts/bench_tool_only.py)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@remigiusz-antczak
remigiusz-antczak merged commit 9665d04 into main Nov 15, 2026
13 checks passed
@remigiusz-antczak
remigiusz-antczak deleted the feat/hub-real-smoke-18 branch November 15, 2026 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant