Repository navigation
Description
Follow-up to #16 and the 11 November evidence correction.
The published Experiment 2 aggregates support arithmetic checks, but the original prompts, responses, events, decisions, run manifests, accounting records, and referenced runs/judge-v1-stopped directory are unavailable in the public repository. The existing Experiment 1 archive contains no Experiment 2 runs.
Recovery and acceptance criteria
- Pio checks the original experiment output directories and relevant backups and records which runs and artifacts exist, including stopped, failed, and incomplete attempts.
- Preserve recovered originals privately without modification. Record file sizes, SHA-256 hashes, source locations, and recovery dates in a private inventory.
- Prepare a sanitized publication copy. Remove credentials, issuer keys, and private data; document transformations and omissions in a manifest linking the public files to their preserved originals. Keep sensitive paths and original inventories private.
- Recompute honest-review counts, attack results, attestations, and coverage from the recovered events. Compare them with the frozen published aggregates and explain every discrepancy.
- Reconcile call totals, costs, retries, completion states, unanswered votes, and the stopped-run account wherever original receipts support doing so. Mark remaining unknowns explicitly.
- Have a second person run the offline reproduction instructions from a fresh checkout and record the source revision, input hashes, commands, and results.
- Publish a dated evidence update without overwriting historical inputs or implying that recovered records were available during the original review.
If recovery is incomplete or impossible, record the search scope, missing artifacts, and outcome. Retain the report's aggregate-only evidence qualification and identify which claims remain independently unverifiable.
This issue covers recovery and offline verification of existing records. It requires no new paid model calls or new credentials. Any rerun must be a separately authorized experiment with its own budget and dataset identity.
Recovery is a follow-up to the accurately qualified correction; missing historical files do not by themselves block merging #16.
Activity
Merge PR #16: gated Experiment 2 and evidence corrections
k2jac9 commentedon Nov 13, 2026
Update from Pio (
IPBin) in his explicit review of #16: he reports that all original Experiment 2 run files are still on his laptop and has checked the scientific corrections against them. He confirmed the DeepSeek honest/attack exceptions, Bandit coverage gaps, 8 harmful-case abstentions, and the incomplete round-4 audit. He also reports 16,416 calls per model, no failed calls or retries, and 225/230 abstentions attributable to length limits. These are presently Pio's source-holder statements, not yet independently reproducible public evidence.He committed to sharing a cleaned set of run files with hashes. The next step under this issue remains: obtain and preserve the originals privately, verify SHA-256 manifests and provenance, independently reproduce every reported count and accounting claim, and publish a sanitized derivation with clear caveats. Do not close this issue until those acceptance criteria are met. #16 has been merged as 5b42448; recovery is separate and still open.
Merge pull request #19 from k2jac9/data/experiment2-runs
Merge pull request #20: verify recovered evidence and runtime behavior
k2jac9 commentedon Nov 16, 2026
Evidence update — 16 November 2026 (AI-assisted, owner-authorized).
PR #20 merged as
e48a75a331b61cfa7f0b861805f485a0e5c47bc5. Its dated recovery verification checks the public archive from #19: 77 data-member hashes, 41,995 retained prompt/event/receipt bindings across 14 runs, recorded accounting, and 46 selected frozen-count comparisons. These checks pass; this establishes internal consistency, not provider authenticity or independently reconciled billing. All twelve runtime lanes passed merged-main CI.The machine-local executable paths in 11 published run manifests now have a separately identified, path-redacted successor proposed in draft PR #21, head
3e99b5de2e9e292bc75fdac3334290bffc7469bd. The original ZIP remains unchanged. The successor SHA-256 isf19127a49dea94e581e3714be3733ba480db7405632f1292b1f737273a23da04; 66 other data members remain byte-identical, andSANITIZATION.jsonmaps all 80 prior members, including three superseded packaging metadata files. The mapping refers to the prior public archive; it does not claim a newly obtained private-original inventory. Existing public history still contains the removed paths.The publication report records scope and limits. Local validation passed 415 Python tests, five viewer tests, the full successor/source comparison, fleet checks, submission verification and runtime freshness. Independent code review found an extra-ZIP-bytes gap; failing regression tests demonstrated it before correction and the full suite passed again.
Human handoff: fresh-checkout instructions and result template. Use the exact head SHA above. An AI rehearsal rebuilt the identical archive from a fresh local clone of the initial candidate; this is not the required second-human reproduction.
Still open:
Keep #17 open. The draft is a reviewable publication follow-up, not closure of the historical evidence gaps. No new paid model calls, experiment reruns or credentials were used.
Hosted follow-up: all twelve runtime lanes passed for
3e99b5de2e9e292bc75fdac3334290bffc7469bdin both the push run and PR run. PR #21 remains a draft for the human evidence handoff; this does not satisfy the pending inventory or human reproduction.k2jac9 commentedon Nov 16, 2026
Status reconciliation — 16 November 2026 (AI-assisted, owner-authorized).
The owner closed #17 at 10:03:31 UTC. That state is preserved; the preceding “keep open” and “draft PR” statements are superseded as repository-status descriptions. PR #21 is now merged at
206fc9997a7de5e7d46753f5e7f33d8fe222c486, and its merged-main CI passed.Closure does not supply new historical evidence. No additional private-original inventory or second-person reproduction receipt has been posted in this issue. The human verification handoff at merged main remains usable. The source-holder inventory, independent human reproduction, missing completion artifacts for three stopped runs, and provider authenticity/billing limitations remain as documented. The sanitized archive and automated consistency checks do not independently establish those facts.
No acceptance boxes have been marked complete on the strength of closure alone. #18 remains open for the unreproduced historical TLS discrepancy; current-environment passes do not identify the cause in the unavailable environment.