Skip to content

Detect transformed fixture canaries in Hermes tool results - #217

Merged
conorbronsdon merged 18 commits into
mainfrom
fix/hermes-transformed-canaries
Sep 27, 2026
Merged

conorbronsdon merged 18 commits into
mainfrom
fix/hermes-transformed-canaries

Conversation

@conorbronsdon

@conorbronsdon conorbronsdon commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Fixes #213. Fixes #219.

The live harness now checks tool-result text for literal fixture canaries, case changes, separators in 32-character hex canaries, and base64/base64url encoding before redacting the result. This restores self-read evidence when a tool transforms the canary. The README describes the bounded detection.

Validation: focused Hermes harness regression and existing self-read tests passed (5 tests); compileall and git diff --check passed. The full harness suite was attempted but did not finish promptly in this environment, so it is not claimed as passing.

Later cycles (see the review records) added wrapped/gzip/nested detection, a bounded decode budget, member allowances, and linear-time matching.

conorbronsdon and others added 9 commits September 25, 2026 12:38
…d on oversize results (#213)

Rejoin wrapped base64 before decoding, inflate gzip payloads under a size
cap, check reversed, percent-encoded, unicode-escaped, and 0x-prefixed
forms, and treat any tool result too large to decode as a self-read
instead of skipping it. Exercise the base64url alphabet in tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…#213)

Replace the backtracking rejoin regex and sub-run combinations with one
linear pass over base64-only lines, trying each group with and without a
stray first or last line. Join a tool result's content fields before
scanning so split parts are checked together, and cap inspected size.
Add linear-time and split, armored, and headed wrapped-base64 tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Replace the per-token cap that flagged large ordinary lines as self-reads
with one decode budget per tool result, inflate every concatenated gzip
member within it, and fail closed only when the budget is exhausted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@conorbronsdon

Copy link
Copy Markdown
Owner Author

Review record

Cycle Reviewer SHA Result
1 Codex gpt-6-astra 9d88918 2 BLOCKING: wrapped base64 split a canary across chunks; tokens over 2 MB were skipped silently. Plus 0x/reversed/gzip/percent/\u gaps, and a base64url test that could not fail.
1 Claude Sonnet 6ebdbb6 2 BLOCKING: the committed test failed (wrapped variant), and the uncommitted rejoin regex backtracked catastrophically (25 s on 40 KB), with quadratic sub-runs.
2 Codex 6ebdbb6 BLOCKING: split-part and BEGIN/END-armored base64 escaped detection; NON_BLOCKING: false red on large plain text.
2 Claude Sonnet c45af2b Both earlier blockers resolved and confirmed by timing (0.008 s where it used to take 25.5 s). No false positives on a 10 MB git log -p.
3 Codex c45af2b 3 BLOCKING: per-token cap false red; later gzip members unscanned; cumulative gzip expansion uncapped.
Final Codex 6fc0565 Those three resolved. 2 BLOCKING remain: the decode budget is double-charged for wrapped lines (a false red on about 14 MB of base64-looking text, which fails safe), and empty gzip members cause unbounded work.

All fixes after the original commit are by Claude. Each has a mutation control that turned the targeted test red and then green again after restoring the code. Validation at 6fc0565: 73 harness tests OK (845 s); Linux validate-all.sh passes; CI runs below.

Merge status: not merged. Three repair cycles are complete. Per the review policy, the remaining findings are filed as #219 rather than fixed in a fourth cycle without approval.

conorbronsdon and others added 9 commits September 26, 2026 15:48
Plain base64 decoding is bounded by the input cap, so it no longer draws on
the decode budget; wrapped lines were charged twice and could flag ordinary
output as a self-read. Cap gzip members per tool result so thousands of
empty members cannot stall the harness, failing closed above the cap.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Decode wrapped base64 groups before individual lines, charge each group's
gzip expansion once by its largest variant, and skip lines already covered
by a group. Independent tokens still draw on the budget one by one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s) (#219)

A misaligned join of unpadded lines can hide a line, so every group line is
also decoded on its own. Both views cover the same bytes, so a group is
charged max(largest variant sum, line sum); tokens within one source are
summed, which also caps many gzip tokens packed into one group.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Replace per-group bookkeeping, which undercounted disjoint variants, with
one running total over every inflation. A payload appears in at most four
decoded views, so the total cap is four times a 10 MB content allowance:
legitimate content is never over-charged, all inflated content is scanned,
and total work stays bounded; exceeding the cap fails closed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ce (#219)

Parse the gzip header and inflate the raw deflate stream in chunks, so a
corrupt CRC or truncated member no longer discards decoded text and member
steps do not copy the whole remaining payload. Scale the member allowance by
the same overlapping-view factor so valid multi-member input seen through
several views is not flagged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…219)

A deflate error, an unreadable header with data remaining, or gzip data
left after the members now fails closed; running out of data mid-member
stays complete. Skip the 8-byte trailer only when the next member lines up,
so a missing trailer cannot hide the following member.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A stored deflate block can carry a following member verbatim, hiding its
canary inside the first member's output. Inflate every gzip signature in
inflated output as a nested member, sharing the budget and member
allowance; nested errors are ignored because the attempts are speculative.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Check reversed markers against each view instead of storing reversed
copies, and strip non-hex characters with a byte-level delete instead of a
regex. A 48 KB result decompressing to 36 MB drops from about 12 s to 0.14 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ature (#219)

Strip the reversed x0 prefix form before the reversed-marker check, and
inflate exactly one member at each nested gzip signature so concatenated
inner members are not re-inflated for every suffix.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@conorbronsdon

Copy link
Copy Markdown
Owner Author

Review record, fourth cycle (approved by the maintainer)

The findings filed as #219 were fixed, and each subsequent re-check was fixed in turn:

Round Reviewer SHA Finding Fix
4a Codex 6fc0565 Double-charged budget for wrapped lines; empty gzip members were unbounded Only gzip inflation is budgeted; per-result member allowance
4b Codex 9f772f8 Wrapped gzip was inflated twice Wrapped groups are decoded first and charged once
4c Codex 73c6ad7 A misaligned unpadded group hid a line; a group's padded tokens were undercharged Every group line is also decoded individually
4d Codex f3621d8 Disjoint variants undercounted One running total, capped at 4 overlapping views × a 10 MB allowance
4e Codex 3808e36 A CRC-corrupt member discarded its output; the member allowance charged overlapping views Raw-deflate inflation in chunks; the member allowance scaled by views
4f Codex 0e2c205 Unreadable, errored, or unfinished members counted as complete; a missing trailer hid the next member Errors and leftover gzip data fail closed; the trailer is skipped only when the next member lines up
4g Codex b3d134a A stored block swallowed the next member Nested gzip signatures inside inflated output are inflated too
4g Claude Sonnet (adversarial) b3d134a CPU: 12 s on 48 KB of input decompressing to 36 MB Byte-level hex stripping; reversed markers checked in place (0.14 s)
4h Codex 1fe4ee2 FOLLOW-UP: reversed 0x dumps; nested suffix re-inflation x0 stripped for the reversed check; one member per nested signature
Final Codex a36577e RESOLVED / NO_FINDINGS —

Every fix has a mutation control: the targeted test goes red with the fix removed or reverted, and green when restored.

Validation at a36577e: 88 harness tests OK (1065 s); Linux validate-all.sh "All validation passed"; CI below.

Remaining documented limit (README): discovery evidence stays bounded against arbitrary transformations. Output containing hundreds of gzip signature bytes fails closed, because it exhausts the member allowance.

@conorbronsdon
conorbronsdon merged commit 07f218f into main Sep 27, 2026
5 checks passed
@conorbronsdon
conorbronsdon deleted the fix/hermes-transformed-canaries branch September 27, 2026 00:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant