|
| 1 | +# How Perun improves itself |
| 2 | + |
| 3 | +**BLUF.** Perun is improved by running Perun on Perun: its own review skill |
| 4 | +(`deep-code-review`), idea attack (`idea-critic`), eval net, and mechanical gates |
| 5 | +are the quality bar for every change to the skillset. This is **Loop 2 of the |
| 6 | +["growing together" model](roadmap.md)** made operational — the runbook, and where |
| 7 | +each piece already lives. This page is a *map*: it links the pieces, it does not |
| 8 | +restate them (the [registry-not-bundle](../CLAUDE.md) thesis, applied to the loop |
| 9 | +itself). |
| 10 | + |
| 11 | +## The loop (each step names the tool that owns it) |
| 12 | + |
| 13 | +| # | Step | What happens | Defined in | |
| 14 | +|---|------|--------------|-----------| |
| 15 | +| 1 | **Signal** | A real-run failure becomes an issue with a *Where / Done-when / Verify / Why* contract — "Verify" is load-bearing, or "done" is an opinion | [roadmap.md](roadmap.md) "Additive principles" | |
| 16 | +| 2 | **Triage + place** | Decide it is worth implementing and general (not slop); assign one home; pre-distinguish it from every existing lens so no second copy is created | [CLAUDE.md](../CLAUDE.md) "No duplication" | |
| 17 | +| 3 | **Draft** | Write the depth as prose in a routed `references/*.md`, with a "Read this when…" trigger; route it from the owning `SKILL.md` | [CLAUDE.md](../CLAUDE.md) "Adding or updating a reference file" | |
| 18 | +| 4 | **Attack** | Run `idea-critic` (skeptic / better-way / kill-criteria) and the advisor *before* substantive work; adopt or rebut each finding against the bytes | [`idea-critic`](../.claude/skills/idea-critic/SKILL.md) | |
| 19 | +| 5 | **Eval** | Add one *behavioral* eval per lens (a prompt and the behavior a correct answer must show) | `.claude/skills/<skill>/evals/evals.json` | |
| 20 | +| 6 | **Review** | An independent dogfood reviewer, held to Perun's own bar, reads the diff for anti-duplication, placement, correctness, privacy, and fabrication — before merge | [roadmap.md](roadmap.md) ("independent reviewer pass before merge"); named as procedure here | |
| 21 | +| 7 | **Gate** | The mechanical gate suite (routing / version / privacy / enumeration / checksums) plus the offline eval-predicate net must all pass | [`ci.yml`](../.github/workflows/ci.yml), [`scripts/ci-gates.sh`](../scripts/ci-gates.sh), [`scripts/eval_predicates.py`](../scripts/eval_predicates.py) | |
| 22 | +| 8 | **Ship** | Lockstep version bump, CHANGELOG, regenerate `SHA256SUMS` last, tag; a human authorizes the landing | the [version gate](../.github/workflows/ci.yml) checks the CHANGELOG announces the version, and [`scripts/write-checksums.sh`](../scripts/write-checksums.sh) regenerates the pin; the full lockstep sequence is maintainer practice, not yet an in-repo doc | |
| 23 | + |
| 24 | +Steps 4–7 are `deep-code-review` and `idea-critic` pointed at Perun's own tree. |
| 25 | +Dogfooding is not a metaphor here — it is the release process. |
| 26 | + |
| 27 | +## The eval rubric is split on purpose — what is built, what is planned |
| 28 | + |
| 29 | +Loop 2 grades improvements on a **split rubric** (see [roadmap.md](roadmap.md)), |
| 30 | +because the two kinds of axis fail differently: |
| 31 | + |
| 32 | +- **Hard axes — deterministic, no model.** Boundary-compliance and non-fabrication |
| 33 | + are graded by code, *because a fluent wrong answer fools an LLM judge*. **Built:** |
| 34 | + `scripts/eval_predicates.py` binds the fabrication-refusal evals to deterministic |
| 35 | + predicates and proves each one discriminates a golden refusal from a golden |
| 36 | + fabrication (`--selftest`, in CI, no API key, no network, no spend). Because the bindings are curated in the |
| 37 | + script rather than in `evals.json`, the layer can change or be removed without |
| 38 | + touching skill content or bumping a version. |
| 39 | +- **Soft axes — LLM-judge, decorrelated.** Whether a *behavioral* lens actually |
| 40 | + changes model behavior needs a model to judge it — the other ~60 evals' behavioral |
| 41 | + expectations are specification today, not executed. **Planned:** issue #61, the |
| 42 | + live eval runner (the "next slice of the eval-harness work" named in |
| 43 | + `scripts/eval_predicates.py`). When it is built it must use a cheap, decorrelated |
| 44 | + judge; enforce a per-run spend cap *before* each call; fail closed on a missing |
| 45 | + key or input; and stay **owner-invoked, never a required CI gate**, because it |
| 46 | + costs money per run. |
| 47 | + |
| 48 | +## Invariants the loop must never break |
| 49 | + |
| 50 | +- **Registry, not a bundle** — link, never duplicate; this page is itself the example. |
| 51 | +- **Quality only ratchets up** — a regression gate stops backsliding, generalized eval |
| 52 | + cases raise the floor, and a human authorizes every landing. |
| 53 | +- **The fuel is verified eval cases and cited standards** — never hoarded user data, |
| 54 | + never a fabricated finding, source, metric, or citation. |
| 55 | +- **Deny by default; unreadable must never look like empty** — a check whose input is |
| 56 | + missing fails closed, and an unverified claim is marked `UNVERIFIED`, not omitted. |
0 commit comments