Skip to content

Commit d991e2f

Browse files
Org audit tools: org_settings_audit, org_audit_driver, org_inventory (#2292)
* Org audit tools: org_settings_audit, org_audit_driver, org_inventory Adds three read-only org-wide tools and their tool-hub route rows. - org_settings_audit.py: audits GitHub organization-level settings (not one repo). - org_audit_driver.py: drives the per-repo audits across an org and records one result file per repo. - org_inventory.py: dated TSV inventory of every repo in one or more orgs; reconciles added and removed repos against a previous run; marks repos without a fresh non-empty `<org>__<repo>.*` result as NOT CHECKED. Fails closed with exit 2 on any API, scope or unexpected error, an empty listing, a count that disagrees with the org totals, a malformed --previous, a non-https --api-url or an off-host Link; follows the Link header with a 1000-page cap and never sends the token to another host. Closes #2278 Closes #2280 Closes #2281 Tests: 50/50 size-budget-raise: .claude/skills/deep-code-review/references/tool-hub.md 17473→19414 org settings audit, org audit driver and org inventory route rows Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * org_audit_driver: real review_checks base, dot-leading repo names, stderr in reasons Fix the stderr-reason assertion in the previous commit (its Tests: 21/21 count was wrong; the suite has 20 checks). The driver now calls review_checks.sh with --full --full-history, allows repo names such as .github (still rejects . and ..), and appends the last stderr line of a failing review_checks to the FAILED reason. Tests: 20/20 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * org_audit_driver: token-scoped private clones, scanner env stripped, 0700 out If ORG_AUDIT_TOKEN is set, the token reaches only the git clone and fetch children, as an http.extraHeader passed through GIT_CONFIG_COUNT/KEY_n/VALUE_n environment entries appended after any the parent already has (never in the URL, argv or .git/config). review_checks and its scanners run with ORG_AUDIT_TOKEN, GH_TOKEN and GITHUB_TOKEN removed and never see the header. The token is masked in FAILED reasons, and --out is created 0700. Verified on a real run of 13 private repositories: all scanned OK, and the token appears in no output file. Tests: 26/26 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Raise agentic-delivery index lookup budget for org_settings_audit row Shorten the org audit script summaries and split their usage paragraphs so the generated index rows stay short. Raise the three index budget rows the regenerated indexes exceed, and reset the lookup-cost cap in test_skill_index_split to the measured lookup plus 100 bytes. Tests: 61/61 size-budget-raise: .claude/skills/INDEX.md 17629→18145 org audit tool rows in the entry index size-budget-raise: .claude/skills/agentic-delivery/INDEX-scripts.md 17639→17829 org_settings_audit row size-budget-raise: .claude/skills/deep-code-review/INDEX-scripts.md 8523→9207 org_audit_driver, org_inventory, org_leak_sweep rows Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 2afdba1 commit d991e2f

22 files changed

Lines changed: 924 additions & 10 deletions

‎.claude/skills/INDEX.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,7 @@ Run `python3 <skill>/scripts/<name>` (`bash` for `.sh`). `name: purpose — argu
7575
- merge_train.py: land several green PRs as ONE verified union (a merge train). — plan --base main [--repo OWNER/NAME]…
7676
- mutation_changed.py: Mutation testing restricted to the lines a diff changed. — [--base origin/main] [--max-seconds…
7777
- operating_selfcheck.py: per-item PRESENT / MISSING / COULD_NOT_CHECK report for the… — [--settings FILE] [--skill-root DIR]
78+
- org_settings_audit.py: read-only audit of GitHub org settings, then each repo's. — ORG [--max-repos N] [--token-env VAR]…
7879
- partial_pass.py: PARTIAL-PASS receipt for a PR whose precheck ended… — write PR --head SHA --unchecked NAME…
7980
- perun_policy.py: read the one resource-policy file, .perun/policy.json. — get <dim>
8081
- pipeline_hub.py: one entry point that says where a delivery pipeline loses… — [--repo OWNER/NAME] [--root DIR]…
@@ -147,6 +148,9 @@ Run `python3 <skill>/scripts/<name>` (`bash` for `.sh`). `name: purpose — argu
147148
- impact_map.py: map a diff's changed symbols to callers and callees. — [--base REF] [--repo DIR] [--cap N]
148149
- lint_ui_slop.py: zero-token, deterministic lint for common UI defects in HTML… — [--json] [--tokens FILE] PATH...
149150
- merge_findings.py: dedupe, rank and cap findings from one or more passes. — [--cap N] [--root DIR] [--ref REF]…
151+
- org_audit_driver.py: review_checks.sh over a repo list, one result file per repo. — --repos FILE --out DIR [--timeout SECS]
152+
- org_inventory.py: dated TSV inventory of GitHub org repos, reconciled with a… — --org ORG [--org ORG ...] [--out DIR]…
153+
- org_leak_sweep.py: scan an org's public repos (or one local repo) for secrets and… — --org ORG [--banlist FILE] [--out DIR]…
150154
- outline.py: cheap symbol map (line ranges) so a lane reads only the ranges… — FILE...
151155
- ownership_sibling_check.py: lead-finder for broken object-level authorization (OWASP API1… — PATH...
152156
- parity_differ.py: the two-sided design<->app parity gate: sections first, then… — --design <file> --app <file> [--accept…

‎.claude/skills/agentic-delivery/INDEX-scripts.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,7 @@ Generated by `scripts/skill_index.py` (do not hand-edit). The script half of `IN
6262
| `scripts/merge_train.py` | land several green PRs as ONE verified union (a merge train). | merge_train.py plan --base main [--repo OWNER/NAME] [--min-checks K] merge_train.py verify --base main --verify-cmd 'ma… |
6363
| `scripts/mutation_changed.py` | Mutation testing restricted to the lines a diff changed. | mutation_changed.py [--base origin/main] [--max-seconds 300] [--strict] [--selftest] |
6464
| `scripts/operating_selfcheck.py` | per-item PRESENT / MISSING / COULD_NOT_CHECK report for the always-on operating layer references/op… | operating_selfcheck.py [--settings FILE] [--skill-root DIR] operating_selfcheck.py --selftest |
65+
| `scripts/org_settings_audit.py` | org_settings_audit.py: read-only audit of GitHub org settings, then each repo's. | org_settings_audit.py ORG [--max-repos N] [--token-env VAR] [--json] |
6566
| `scripts/outcome_claim_guard.py` | Stop + PreToolUse + UserPromptSubmit hook: no "fixed / faster / done" on delivery outcomes without… | — |
6667
| `scripts/partial_pass.py` | PARTIAL-PASS receipt for a PR whose precheck ended COULD_NOT_CHECK on some checks (#1778). | partial_pass.py write PR --head SHA --unchecked NAME [NAME ...] |
6768
| `scripts/peer_ack.py` | read receipts for peer directives, on the coordination board: no ack within T seconds means undeliv… | — |
@@ -84,7 +85,7 @@ Generated by `scripts/skill_index.py` (do not hand-edit). The script half of `IN
8485
| `scripts/refix_gate.py` | re-touching a recently-fixed file requires a class-level artifact. | refix_gate.py --base REF --head REF [--window-hours 72] [--class-glob GLOB ...] refix_gate.py --selftest |
8586
| `scripts/repeat_poll_guard.py` | PreToolUse(Bash) hook: nudge when the same command runs more than 3 times in one session. | — |
8687
| `scripts/reply_cap.py` | owner rule: chat replies max 3 lines, 500 chars. | — |
87-
| `scripts/repo_settings_audit.py` | repo_settings_audit.py: read-only audit of the GitHub settings that decide who can change a reposit… | repo_settings_audit.py OWNER/NAME [OWNER/NAME ...] [--repos-file FILE] [--json] --repos-file holds one OWNER/NAME per l… |
88+
| `scripts/repo_settings_audit.py` | repo_settings_audit.py: read-only audit of the GitHub settings that decide who can change a reposit… | repo_settings_audit.py OWNER/NAME [OWNER/NAME ...] [--repos-file FILE] [--json] [--scorecard] --repos-file holds one OW… |
8889
| `scripts/restore_ledger.py` | classify every branch after a VCS history-rewrite incident. | — |
8990
| `scripts/review_digest.py` | a returning reviewer's change digest, not a full re-read. | review_digest.py --since SHA [--head REF] [--paths PATH ...] [--max-lines N] [--allow-diverged] review_digest.py --self… |
9091
| `scripts/review_gate.py` | pre-merge gate: every PR needs an independent deep-code-review DIFF pass plus an evil-twin check. | review_gate.py check <pr> --head SHA [--repo R] |

‎.claude/skills/agentic-delivery/INDEX.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,4 +38,4 @@ Generated by `scripts/skill_index.py` (do not hand-edit; rerun it after changing
3838

3939
## Scripts
4040

41-
110 scripts. Read `INDEX-scripts.md` when you need a script (a trigger and CLI per script). `../INDEX.md`, `### agentic-delivery`, has only each one-line purpose and arguments.
41+
111 scripts. Read `INDEX-scripts.md` when you need a script (a trigger and CLI per script). `../INDEX.md`, `### agentic-delivery`, has only each one-line purpose and arguments.
Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
#!/usr/bin/env python3
2+
"""org_settings_audit.py: read-only audit of GitHub org settings, then each repo's.
3+
4+
WHY: branch protection, required 2FA, Actions token defaults, installed apps and self-hosted runners are not visible in
5+
code, yet they decide whether a review gate means anything (the CIS GitHub benchmark covers them). Per repository this
6+
reuses `repo_settings_audit.audit` unchanged (branch protection and rulesets, Actions, collaborators, deploy keys,
7+
webhooks, environments, secret scanning and push protection); this file adds the organization-level checks:
8+
2FA `two_factor_requirement_enabled` on the organization (owner token only)
9+
Actions organization default workflow token has write, workflows can approve reviews, allowed actions = all
10+
apps installed GitHub Apps holding admin or write on repository administration or contents (count)
11+
outside people outside collaborators (count)
12+
self-hosted organization-level self-hosted runners (count)
13+
A 403 or 404, a missing field or an odd reply is a `not checked` row that names the token scope or permission it needs,
14+
never a pass.
15+
16+
USAGE: org_settings_audit.py ORG [--max-repos N] [--token-env VAR] [--json]
17+
18+
Token: the env var named by --token-env, else `gh auth token`; it reaches curl on stdin only (never argv), as in
19+
org_leak_sweep.py. `GITHUB_API_URL` overrides https://api.github.com (tests point it at a stub).
20+
Output: `{"org": rows, "repos": {"OWNER/NAME": rows}, "summary": {...}}` with `--json`, else one table.
21+
Exit codes: 0 the audit ran (findings do not change it), 2 usage or no token.
22+
Side effects: HTTP GET only; nothing is written or changed. Counts and booleans only: no login, app name or API text.
23+
Limits: first --max-repos repositories (default 200); a longer org says so in a not-checked row.
24+
"""
25+
from __future__ import annotations
26+
27+
import argparse
28+
import json
29+
import os
30+
import re
31+
import subprocess
32+
import sys
33+
import urllib.parse
34+
35+
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
36+
import repo_settings_audit as rsa # noqa: E402
37+
38+
API = "https://api.github.com"
39+
ORG = re.compile(r"[A-Za-z0-9][A-Za-z0-9-]*")
40+
41+
42+
def curl_get(token):
43+
"""get(path, params=None) -> parsed JSON over curl; failure raises OSError('... (HTTP n)') for `rsa.why`."""
44+
def get(path, params=None):
45+
url = os.environ.get("GITHUB_API_URL", API) + path + ("?" + urllib.parse.urlencode(params) if params else "")
46+
cfg = 'header = "Authorization: Bearer ' + token.replace("\\", "\\\\").replace('"', '\\"') + '"\n'
47+
r = subprocess.run(["curl", "-sS", "-w", "\n%{http_code}", "-H", "Accept: application/vnd.github+json",
48+
"--config", "-", url], input=cfg, capture_output=True, text=True, errors="replace")
49+
body, _, code = r.stdout.rpartition("\n")
50+
if r.returncode or not code.isdigit():
51+
raise OSError("API error")
52+
if int(code) >= 400:
53+
try:
54+
msg = str(json.loads(body).get("message", ""))
55+
except (ValueError, AttributeError):
56+
msg = ""
57+
raise OSError("%s (HTTP %s)" % (msg, code))
58+
return json.loads(body)
59+
return get
60+
61+
62+
def need(scope):
63+
return ("needs " + scope, "re-run with a token that has " + scope)
64+
65+
66+
def org_rows(get, org):
67+
"""(check, severity, fix) tuples for the organization itself; any failed read is a not-checked row."""
68+
base, rows = "/orgs/" + org, []
69+
o, e = rsa.read(get, base, dict)
70+
if e:
71+
return [rsa.nc("organization", rsa.why(e, need("read access to the organization")))]
72+
tf = o.get("two_factor_requirement_enabled") # absent unless the token is an org owner
73+
if tf is None:
74+
rows.append(rsa.nc("2FA requirement", need("owner access (admin:org)")))
75+
elif tf is not True:
76+
rows.append(("2FA requirement is off", "high", "enable 'Require two-factor authentication' in the organization security settings"))
77+
adm = "admin:org (fine-grained: Administration read)"
78+
w, e = rsa.read(get, base + "/actions/permissions/workflow", dict,
79+
need=(("default_workflow_permissions",), ("can_approve_pull_request_reviews",)))
80+
if e:
81+
rows.append(rsa.nc("org Actions token", rsa.why(e, need(adm))))
82+
else:
83+
if w["default_workflow_permissions"] == "write":
84+
rows.append(("org default workflow token can write", "medium", "set the organization default workflow permissions to read"))
85+
if w["can_approve_pull_request_reviews"] is True:
86+
rows.append(("org workflows can approve pull request reviews", "medium", "turn off 'Allow GitHub Actions to create and approve pull requests'"))
87+
a, e = rsa.read(get, base + "/actions/permissions", dict, need=(("enabled_repositories",),))
88+
if e:
89+
rows.append(rsa.nc("org allowed actions", rsa.why(e, need(adm))))
90+
elif a.get("allowed_actions") == "all":
91+
rows.append(("org allows all actions", "medium", "restrict allowed actions to GitHub-owned and an allowlist"))
92+
ins, e = rsa.read(get, base + "/installations", dict, need=(("installations",),))
93+
if e:
94+
rows.append(rsa.nc("installed apps", rsa.why(e, need("admin:read (fine-grained: Administration read)"))))
95+
else:
96+
n = sum(1 for i in ins["installations"] if isinstance(i, dict) and any(
97+
(i.get("permissions") or {}).get(k) in ("write", "admin") for k in ("administration", "contents")))
98+
rows.append(("%d installed app(s) hold write or admin on administration or contents" % n, "low" if n else "info",
99+
"review each app's permissions in the organization installed apps settings"))
100+
oc, e = rsa.listing(get, base + "/outside_collaborators")
101+
rows.append(rsa.nc("outside collaborators", rsa.why(e, need("admin:org (owner)"))) if e else
102+
("%d outside collaborator(s)" % len(oc), "low" if oc else "info", "confirm each still needs access"))
103+
rn, e = rsa.listing(get, base + "/actions/runners", key="runners")
104+
rows.append(rsa.nc("self-hosted runners", rsa.why(e, need("admin:org (fine-grained: Self-hosted runners read)"))) if e else
105+
("%d org self-hosted runner(s)" % len(rn), "medium" if rn else "info",
106+
"keep self-hosted runners off public repositories and use ephemeral runners"))
107+
return rows
108+
109+
110+
def audit(org, get, max_repos=200):
111+
"""(org rows, {slug: rows}); an unreadable or over-long repo list is a not-checked org row."""
112+
keys = ("check", "severity", "fix")
113+
rows = [dict(zip(keys, r)) for r in org_rows(get, org)]
114+
repos, e = rsa.listing(get, "/orgs/%s/repos" % org, {"type": "all"}, need=(("full_name",),))
115+
if e:
116+
rows.append(dict(zip(keys, rsa.nc("repository list", rsa.why(e, need("read access to the organization's repositories"))))))
117+
repos = []
118+
if len(repos) > max_repos:
119+
rows.append(dict(zip(keys, rsa.nc("repository list", ("capped at %d repos" % max_repos, "raise --max-repos")))))
120+
rows.sort(key=lambda r: rsa.SEVERITY.index(r["severity"]))
121+
return rows, {r["full_name"]: rsa.audit(r["full_name"], get) for r in repos[:max_repos]}
122+
123+
124+
def main(argv=None, get=None):
125+
ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
126+
ap.add_argument("org")
127+
ap.add_argument("--max-repos", type=int, default=200)
128+
ap.add_argument("--token-env")
129+
ap.add_argument("--json", action="store_true")
130+
a = ap.parse_args(argv)
131+
if not ORG.fullmatch(a.org):
132+
print("org_settings_audit: not an organization name", file=sys.stderr)
133+
return 2
134+
if get is None:
135+
tok = os.environ.get(a.token_env, "") if a.token_env else subprocess.run(
136+
["gh", "auth", "token"], capture_output=True, text=True).stdout.strip()
137+
if not tok:
138+
print("org_settings_audit: no token (set --token-env VAR or run `gh auth login`)", file=sys.stderr)
139+
return 2
140+
get = curl_get(tok)
141+
org, repos = audit(a.org, get, a.max_repos)
142+
flat = [r for v in repos.values() for r in v]
143+
summ = {"repos": len(repos), "findings": sum(r["severity"] in rsa.FINDING for r in org + flat),
144+
"not_checked": sum(r["severity"] == "not-checked" for r in org + flat)}
145+
if a.json:
146+
print(json.dumps({"org": org, "repos": repos, "summary": summ}, indent=2))
147+
else:
148+
print(rsa.render([dict(r, repo=a.org) for r in org] + flat, len(repos))[0])
149+
return 0
150+
151+
152+
if __name__ == "__main__":
153+
sys.exit(main())

‎.claude/skills/deep-code-review/INDEX-scripts.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,9 @@ Generated by `scripts/skill_index.py` (do not hand-edit). The script half of `IN
3131
| `scripts/invisible_chars.py` | flag NUL, control and hidden characters in text files. | — |
3232
| `scripts/lint_ui_slop.py` | zero-token, deterministic lint for common UI defects in HTML, CSS, JSX and TSX. | lint_ui_slop.py [--json] [--tokens FILE] PATH... lint_ui_slop.py --hook < hook-event.json |
3333
| `scripts/merge_findings.py` | dedupe, rank and cap findings from one or more passes. | merge_findings.py [--cap N] [--root DIR] [--ref REF] <findings.json>... |
34+
| `scripts/org_audit_driver.py` | org_audit_driver.py: review_checks.sh over a repo list, one result file per repo. | org_audit_driver.py --repos FILE --out DIR [--timeout SECS] |
35+
| `scripts/org_inventory.py` | org_inventory.py: dated TSV inventory of GitHub org repos, reconciled with a previous run. | org_inventory.py --org ORG [--org ORG ...] [--out DIR] [--previous TSV] [--results-dir DIR] [--max-age-days N] [--token… |
36+
| `scripts/org_leak_sweep.py` | org_leak_sweep.py: scan an org's public repos (or one local repo) for secrets and banned terms. | org_leak_sweep.py --org ORG [--banlist FILE] [--out DIR] [--token-env VAR] org_leak_sweep.py --local REPO [--banlist FI… |
3437
| `scripts/outline.py` | cheap symbol map (line ranges) so a lane reads only the ranges it needs, not the whole file. | outline.py FILE... |
3538
| `scripts/ownership_sibling_check.py` | lead-finder for broken object-level authorization (OWASP API1, CWE-639) by sibling comparison. | ownership_sibling_check.py PATH... |
3639
| `scripts/parity_differ.py` | the two-sided design<->app parity gate: sections first, then the element inventory of every matched… | parity_differ.py --design <file> --app <file> [--accept <tsv> [--accept-rev REV]] [--min-pairs N] [--style [--style-min… |

‎.claude/skills/deep-code-review/INDEX.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -138,7 +138,7 @@ Generated by `scripts/skill_index.py` (do not hand-edit; rerun it after changing
138138
| `references/testing-ui.md` | the target renders a UI or ships browser / E2E specs. | 4868 |
139139
| `references/threat-modeling.md` | a change adds a trust boundary, an auth path, a data flow to a new party, an agent or tool capabili… | 2574 |
140140
| `references/time-date-correctness.md` | reviewing anything that stores, computes, schedules, or renders timestamps, durations, recurring ev… | 4809 |
141-
| `references/tool-hub.md` | a review should lean on external scanners via the tool hub (workflows, Python, Go, SBOM/lockfiles,… | 3942 |
141+
| `references/tool-hub.md` | a review should lean on external scanners via the tool hub (workflows, Python, Go, SBOM/lockfiles,… | 4853 |
142142
| `references/ux-components.md` | the target or diff renders one concept in more than one place: a shared component or design token,… | 7235 |
143143
| `references/ux-dataviz.md` | the target or diff renders a chart, sparkline, heat-map or calendar grid, a KPI / metric delta, a c… | 2787 |
144144
| `references/ux-gates.md` | a UI change is claimed verified on screenshot evidence, the project's auto-merge or CI policy gates… | 3756 |
@@ -153,4 +153,4 @@ Generated by `scripts/skill_index.py` (do not hand-edit; rerun it after changing
153153

154154
## Scripts
155155

156-
48 scripts. Read `INDEX-scripts.md` when you need a script (a trigger and CLI per script). `../INDEX.md`, `### deep-code-review`, has only each one-line purpose and arguments.
156+
51 scripts. Read `INDEX-scripts.md` when you need a script (a trigger and CLI per script). `../INDEX.md`, `### deep-code-review`, has only each one-line purpose and arguments.

0 commit comments

Comments
 (0)