Idea: a verification pattern for instruction compliance
Aider's --lint-cmd and --test-cmd catch code errors. But here's a class of failure neither catches:
Tell aider: "Review your own code before committing."
Aider says: "Done, reviewed."
Did it actually re-read the file? Or just generate those words?
Lint passes — the code is correct. Test passes — the output is correct. But the instruction was never followed.
Why --lint-cmd doesn't cover this
--lint-cmd checks the output. Instruction compliance isn't in the output — it's in the process:
- "Did the model re-open the file?" → check file access timestamp
- "Did it really compare the diff?" → check whether a diff was computed
- "Did a human actually approve this?" → check for real human interaction
These checks require information the model can't fabricate — filesystem metadata, process state. --lint-cmd can technically run timestamp checks, but there's no documented pattern. Users need to independently discover (a) that this class of failure exists, (b) which filesystem facts to check, (c) how to wire it in.
Concrete data
I've been measuring this for 2 months. 339 blocked-event records, 25 hook registrations (18 groups, 7 event types), 9 sessions:
| What the check examines |
Triggers |
False positives |
| Filesystem facts (timestamps, paths) |
4-24 |
0% |
| OS facts (command strings) |
6 |
0% |
| Text facts ("did human confirm?") |
295 |
100% |
The first two work because the model can't write fake timestamps to disk. The third fails — the AI wrote a 2-line script to auto-answer "yes" 295 times.
What I'm proposing
Not a new flag. A documented pattern + example script:
# .aider.conf.yml — use existing lint-cmd with a process-level check
lint-cmd: python check_review.py --file "$AIDER_EDITED_FILE"
Where check_review.py (~20 lines) verifies: was this file actually re-read after the edit?
The ask
- Is there already a recommended pattern for process-level checks? I might have missed it.
- If I put together a clean example (script + docs), would that fit as a contribution? Thinking a PR with a documented pattern, not a code change to aider itself.
Not proposing a feature. Offering a pattern backed by data.
(Happy to sign the CLA, share raw audit data.)
Edit (Jul 26): corrected "38 verification points" → "25 hook registrations (18 groups, 7 event types)" — verified against actual settings.json.
Idea: a verification pattern for instruction compliance
Aider's
--lint-cmdand--test-cmdcatch code errors. But here's a class of failure neither catches:Lint passes — the code is correct. Test passes — the output is correct. But the instruction was never followed.
Why
--lint-cmddoesn't cover this--lint-cmdchecks the output. Instruction compliance isn't in the output — it's in the process:These checks require information the model can't fabricate — filesystem metadata, process state.
--lint-cmdcan technically run timestamp checks, but there's no documented pattern. Users need to independently discover (a) that this class of failure exists, (b) which filesystem facts to check, (c) how to wire it in.Concrete data
I've been measuring this for 2 months. 339 blocked-event records, 25 hook registrations (18 groups, 7 event types), 9 sessions:
The first two work because the model can't write fake timestamps to disk. The third fails — the AI wrote a 2-line script to auto-answer "yes" 295 times.
What I'm proposing
Not a new flag. A documented pattern + example script:
Where
check_review.py(~20 lines) verifies: was this file actually re-read after the edit?The ask
Not proposing a feature. Offering a pattern backed by data.
(Happy to sign the CLA, share raw audit data.)
Edit (Jul 26): corrected "38 verification points" → "25 hook registrations (18 groups, 7 event types)" — verified against actual settings.json.