Skip to content

Commit d55df1d

Browse files
remigiusz-antczakRemigiusz Antczakclaude
authored
docs: add the self-improvement loop runbook (docs/self-improvement.md) (#173)
Package the end-to-end loop by which Perun improves itself — the 8-step procedure (signal -> triage -> draft -> attack -> eval -> review -> gate -> ship), the tool each step owns, and the built-vs-planned state of the split eval rubric — as one routed map that links the existing pieces and restates none of them (the registry-not-bundle thesis applied to the loop itself). - docs/self-improvement.md (new): the runbook plus the split-rubric state — offline fabrication predicates built; the #61 LLM-judge live runner planned, with its spend-cap / fail-closed / owner-invoked constraints stated. - Pointers from README "Trust & safety" and docs/roadmap.md "growing together". - CONTRIBUTING.md: complete the pre-commit gate list (add the enumeration gate and the eval-predicate selftest, both already run by ci.yml). Docs-only; no skill content changed, no version bump. Ran the loop on this change itself: the advisor (step 4) caught a stale CONTRIBUTING gate list and an overclaim; the independent dogfood reviewer (step 6) caught a step-8 citation error — both fixed before merge. The loop, applied to its own documentation. Co-authored-by: Remigiusz Antczak <remigiusz@antczak.io> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 3a8f49c commit d55df1d

4 files changed

Lines changed: 65 additions & 1 deletion

File tree

‎CONTRIBUTING.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,8 @@ bash scripts/ci-gates.sh routing --max-bytes 24000 .claude/skills/business-ops
3939
bash scripts/ci-gates.sh routing --max-bytes 24000 .claude/skills/product-output-safety
4040
bash scripts/ci-gates.sh version .
4141
bash scripts/ci-gates.sh privacy --banlist .banlist.txt .
42+
bash scripts/ci-gates.sh enumeration .
43+
python3 scripts/eval_predicates.py --selftest
4244
bash -n install.sh
4345
```
4446

‎README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -261,7 +261,9 @@ identical API/LLM/DB calls, over-fetching, "call it every run" patterns).
261261

262262
## Trust & safety
263263

264-
Perun is built to be trusted with your code, and it dogfoods its own rules.
264+
Perun is built to be trusted with your code, and it dogfoods its own rules — the
265+
end-to-end loop it uses to improve itself is written down in
266+
[docs/self-improvement.md](docs/self-improvement.md).
265267

266268
- **No fabrication.** Skip or flag rather than guess — no invented findings, data,
267269
sources, metrics, CWEs, or line numbers.

‎docs/roadmap.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,10 @@ up**: a regression gate stops backsliding, generalized eval cases raise the floo
152152
and a human authorizes every landing. The fuel is verified eval cases and
153153
citations — not hoarded user data.
154154

155+
**Operational runbook:** [How Perun improves itself](self-improvement.md) makes this
156+
loop executable — the concrete 8 steps, the tool each one uses, and the built-vs-planned
157+
state of the split rubric.
158+
155159
## Additive principles (generalized from field practice and research)
156160
- **Registry, not a bundle** — describe how to reach things; never duplicate them.
157161
- **Store the query, not the answer; audit memory automatically** — a stored status

‎docs/self-improvement.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
# How Perun improves itself
2+
3+
**BLUF.** Perun is improved by running Perun on Perun: its own review skill
4+
(`deep-code-review`), idea attack (`idea-critic`), eval net, and mechanical gates
5+
are the quality bar for every change to the skillset. This is **Loop 2 of the
6+
["growing together" model](roadmap.md)** made operational — the runbook, and where
7+
each piece already lives. This page is a *map*: it links the pieces, it does not
8+
restate them (the [registry-not-bundle](../CLAUDE.md) thesis, applied to the loop
9+
itself).
10+
11+
## The loop (each step names the tool that owns it)
12+
13+
| # | Step | What happens | Defined in |
14+
|---|------|--------------|-----------|
15+
| 1 | **Signal** | A real-run failure becomes an issue with a *Where / Done-when / Verify / Why* contract — "Verify" is load-bearing, or "done" is an opinion | [roadmap.md](roadmap.md) "Additive principles" |
16+
| 2 | **Triage + place** | Decide it is worth implementing and general (not slop); assign one home; pre-distinguish it from every existing lens so no second copy is created | [CLAUDE.md](../CLAUDE.md) "No duplication" |
17+
| 3 | **Draft** | Write the depth as prose in a routed `references/*.md`, with a "Read this when…" trigger; route it from the owning `SKILL.md` | [CLAUDE.md](../CLAUDE.md) "Adding or updating a reference file" |
18+
| 4 | **Attack** | Run `idea-critic` (skeptic / better-way / kill-criteria) and the advisor *before* substantive work; adopt or rebut each finding against the bytes | [`idea-critic`](../.claude/skills/idea-critic/SKILL.md) |
19+
| 5 | **Eval** | Add one *behavioral* eval per lens (a prompt and the behavior a correct answer must show) | `.claude/skills/<skill>/evals/evals.json` |
20+
| 6 | **Review** | An independent dogfood reviewer, held to Perun's own bar, reads the diff for anti-duplication, placement, correctness, privacy, and fabrication — before merge | [roadmap.md](roadmap.md) ("independent reviewer pass before merge"); named as procedure here |
21+
| 7 | **Gate** | The mechanical gate suite (routing / version / privacy / enumeration / checksums) plus the offline eval-predicate net must all pass | [`ci.yml`](../.github/workflows/ci.yml), [`scripts/ci-gates.sh`](../scripts/ci-gates.sh), [`scripts/eval_predicates.py`](../scripts/eval_predicates.py) |
22+
| 8 | **Ship** | Lockstep version bump, CHANGELOG, regenerate `SHA256SUMS` last, tag; a human authorizes the landing | the [version gate](../.github/workflows/ci.yml) checks the CHANGELOG announces the version, and [`scripts/write-checksums.sh`](../scripts/write-checksums.sh) regenerates the pin; the full lockstep sequence is maintainer practice, not yet an in-repo doc |
23+
24+
Steps 4–7 are `deep-code-review` and `idea-critic` pointed at Perun's own tree.
25+
Dogfooding is not a metaphor here — it is the release process.
26+
27+
## The eval rubric is split on purpose — what is built, what is planned
28+
29+
Loop 2 grades improvements on a **split rubric** (see [roadmap.md](roadmap.md)),
30+
because the two kinds of axis fail differently:
31+
32+
- **Hard axes — deterministic, no model.** Boundary-compliance and non-fabrication
33+
are graded by code, *because a fluent wrong answer fools an LLM judge*. **Built:**
34+
`scripts/eval_predicates.py` binds the fabrication-refusal evals to deterministic
35+
predicates and proves each one discriminates a golden refusal from a golden
36+
fabrication (`--selftest`, in CI, no API key, no network, no spend). Because the bindings are curated in the
37+
script rather than in `evals.json`, the layer can change or be removed without
38+
touching skill content or bumping a version.
39+
- **Soft axes — LLM-judge, decorrelated.** Whether a *behavioral* lens actually
40+
changes model behavior needs a model to judge it — the other ~60 evals' behavioral
41+
expectations are specification today, not executed. **Planned:** issue #61, the
42+
live eval runner (the "next slice of the eval-harness work" named in
43+
`scripts/eval_predicates.py`). When it is built it must use a cheap, decorrelated
44+
judge; enforce a per-run spend cap *before* each call; fail closed on a missing
45+
key or input; and stay **owner-invoked, never a required CI gate**, because it
46+
costs money per run.
47+
48+
## Invariants the loop must never break
49+
50+
- **Registry, not a bundle** — link, never duplicate; this page is itself the example.
51+
- **Quality only ratchets up** — a regression gate stops backsliding, generalized eval
52+
cases raise the floor, and a human authorizes every landing.
53+
- **The fuel is verified eval cases and cited standards** — never hoarded user data,
54+
never a fabricated finding, source, metric, or citation.
55+
- **Deny by default; unreadable must never look like empty** — a check whose input is
56+
missing fails closed, and an unverified claim is marked `UNVERIFIED`, not omitted.

0 commit comments

Comments
 (0)