Skip to content

Bench: seed six web CWE classes and harden the Python taint guards - #2386

Merged
remigiusz-antczak merged 8 commits into
mainfrom
feat/bench-wave19
Nov 15, 2026
Merged

remigiusz-antczak merged 8 commits into
mainfrom
feat/bench-wave19

Conversation

@remigiusz-antczak

Copy link
Copy Markdown
Collaborator

The tool-only bench now seeds CWE-79 (Express reflected XSS), CWE-918 (SSRF), CWE-611 (lxml XXE), CWE-601 (open redirect), CWE-295 (requests verify=False) and CWE-347 (PyJWT signature verification disabled), each with a clean twin, and the answer key lists all six.

Rule changes in the Python Semgrep pack:

  • SSRF is suppressed only when the tainted variable is the direct URL argument behind an allowlist abort guard; an f-string URL still fires.
  • Open redirect skips relative paths, but a leading "//" or "/" still fires.
  • New rule perun.py.xxe-lxml flags resolve_entities=True, load_dtd=True and huge_tree=True on lxml parsers.
  • The "not in" allowlist guard accepts a non-literal container.

New test scripts/test_semgrep_py_guards.py (11 ruleid cases, 6 ok cases) is wired through scripts/ci-tests.d/semgrep-py-guards.sh.

Tests: 3/3 (test_semgrep_py_guards, test_semgrep_py_pathtrav, test_semgrep_defaults; local bench recall 1.0 on every new class)
🤖 Generated with Claude Code

remigiusz-antczak and others added 8 commits November 15, 2026 18:51
… ceiling to 26

The CI real-hub job measured 11/11 recall and 26 false positives on the clean twins after the CWE-22 taint rule stopped firing on send_from_directory's file name. Both values are now frozen so a regression in any seeded class fails CI.

Tests: 2/2 (py_compile; CI real-hub job on the merged rule measured recall 1.0, 26 false positives)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds CWE-79, 918, 611, 601, 295 and 347 keys; fixtures follow in the next commit.

Tests: n/a (answer key only; scored in the next commit)
No-Mechanism-Reason: answer key data, no skill prose changed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fictional XSS, SSRF, XXE, open-redirect, TLS-verify and JWT-verify fixtures. Local run without the full hub: 14/17 recall, 15 false positives; CWE-918, 611 and 601 missed locally because their rules also fire on the clean twins.

Tests: 1/1 (REQUIRE_REAL_HUB=0 bench_tool_only.py run locally; FLOOR_RECALL unchanged, CI measures)
No-Mechanism-Reason: bench fixtures only, no skill prose changed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…xml XXE rule

perun.py.ssrf skips sinks after `if url not in ALLOWED: abort(...)`.
perun.py.open-redirect skips sinks after the startswith("/")/("//") reassign guard
and after url_has_allowed_host_and_scheme guards. New perun.py.xxe-lxml (CWE-611)
flags lxml XMLParser(resolve_entities=True) and no_network=False.

Tests: 3/3 (test_semgrep_py_guards, test_semgrep_py_pathtrav, test_semgrep_defaults)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SSRF guard now requires the guarded value to be the URL argument itself and the
container not to be a string literal. Redirect guard also requires a "/\" check.
xxe-lxml also flags load_dtd=True and huge_tree=True. New negative fixtures and
offline pins for each.

Tests: 3/3 (test_semgrep_py_guards, test_semgrep_py_pathtrav, test_semgrep_defaults)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng at 31

CI real-hub flagged ruff I001 and PIE810 on the new clean ssrf/redirect twins; fix them and set the ceiling to 31 for the 5 remaining registry/semgrep FPs on guarded code.

Tests: 2/2 (ruff --select I001,PIE810 on both twins clean; bench_tool_only locally FP 13 <= 31, local recall 0.941 reflects missing semgrep, CI real-hub is authoritative)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ling at 32

The tuple rewrite of the redirect clean twin broke its guard, so the real-hub job missed CWE-601 and flagged a new false positive.

Tests: 1/1 (py_compile scripts/bench_tool_only.py)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@remigiusz-antczak
remigiusz-antczak merged commit 9f9b891 into main Nov 15, 2026
13 checks passed
@remigiusz-antczak
remigiusz-antczak deleted the feat/bench-wave19 branch November 15, 2026 19:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant