Repository navigation
Report lint sink rules; G6 set2 result - #2230
Merged
Merged
Conversation
…scored result report_contract_lint.py gains two rules aimed at the false positives found when blind data/API set2 was scored and adjudicated. Most of those false positives cited the route or helper line instead of the sink, or used the wrong class. finding-not-on-sink flags a Findings row for an outbound class (ssrf, outbound-timeout, tls-off, secret-in-url), a log class (secret-in-log, pii-in-log) or cookie-flags when no line within two lines of the cite shows the matching sink. class-mismatch flags a pii-in-log row in two cases: when it logs a credential-named value and no personal-data word (use secret-in-log), and when the line is an outbound call with no log call nearby (use pii-data). whole-project-report.md now tells the reviewer to cite the sink line. This also lands the set2 blind fixtures and the scored, adjudicated result: FAIL, 3 of 8 parts. Recall passes in all 8 parts and every failure is precision. Set2 is burned. Tests: 89/89 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two report_contract_lint.py rules aimed at the false positives found when a blind data/API set was scored and adjudicated. Most of those false positives cited the route or helper line instead of the sink, or used the wrong class.
finding-not-on-sink: a Findings row for an outbound class (ssrf, outbound-timeout, tls-off, secret-in-url), a log class (secret-in-log, pii-in-log) or cookie-flags is flagged when no line within two lines of the cite shows the matching sink. The sinks are an outbound call, URL or TLS option; a log or print call; or a cookie or session config.class-mismatch: a pii-in-log row is flagged when it logs a credential-named value and no personal-data word (use secret-in-log). It is also flagged when the cited line is an outbound call with no log call nearby (use pii-data).whole-project-report.md now tells the reviewer to cite the sink line. One existing fixture changed its class from SSRF to XSS so it keeps testing the rule it was written for.
This PR also lands the set2 blind fixtures (
docs/bench/g6-data-api/set2/) and the scored, adjudicated result: FAIL, 3 of 8 parts. Recall passes in all 8 parts and every failure is precision. Set2 is burned. The raw and adjudicatedscore.json, the adjudication input and verdicts, and the scored event inledger.jsonlare included.Tests 89/89: lint 48 (41 existing plus 7 new sink and class tests), bench 41. Privacy, size-ratchet, changelog-fragment and SHA256SUMS gates pass locally. An independent review was done and its folds verified.
🤖 Generated with Claude Code