Skip to content

Recover and verify the original Experiment 2 evidence #17

Closed

Description

@k2jac9

Follow-up to #16 and the 11 November evidence correction.

The published Experiment 2 aggregates support arithmetic checks, but the original prompts, responses, events, decisions, run manifests, accounting records, and referenced runs/judge-v1-stopped directory are unavailable in the public repository. The existing Experiment 1 archive contains no Experiment 2 runs.

Recovery and acceptance criteria

  • Pio checks the original experiment output directories and relevant backups and records which runs and artifacts exist, including stopped, failed, and incomplete attempts.
  • Preserve recovered originals privately without modification. Record file sizes, SHA-256 hashes, source locations, and recovery dates in a private inventory.
  • Prepare a sanitized publication copy. Remove credentials, issuer keys, and private data; document transformations and omissions in a manifest linking the public files to their preserved originals. Keep sensitive paths and original inventories private.
  • Recompute honest-review counts, attack results, attestations, and coverage from the recovered events. Compare them with the frozen published aggregates and explain every discrepancy.
  • Reconcile call totals, costs, retries, completion states, unanswered votes, and the stopped-run account wherever original receipts support doing so. Mark remaining unknowns explicitly.
  • Have a second person run the offline reproduction instructions from a fresh checkout and record the source revision, input hashes, commands, and results.
  • Publish a dated evidence update without overwriting historical inputs or implying that recovered records were available during the original review.

If recovery is incomplete or impossible, record the search scope, missing artifacts, and outcome. Retain the report's aggregate-only evidence qualification and identify which claims remain independently unverifiable.

This issue covers recovery and offline verification of existing records. It requires no new paid model calls or new credentials. Any rerun must be a separately authorized experiment with its own budget and dataset identity.

Recovery is a follow-up to the accurately qualified correction; missing historical files do not by themselves block merging #16.

Activity

  1. added a commit that references this issue on Nov 13, 2026
  2. k2jac9 commented on Nov 13, 2026

    @k2jac9
    OwnerAuthor

    Update from Pio (IPBin) in his explicit review of #16: he reports that all original Experiment 2 run files are still on his laptop and has checked the scientific corrections against them. He confirmed the DeepSeek honest/attack exceptions, Bandit coverage gaps, 8 harmful-case abstentions, and the incomplete round-4 audit. He also reports 16,416 calls per model, no failed calls or retries, and 225/230 abstentions attributable to length limits. These are presently Pio's source-holder statements, not yet independently reproducible public evidence.

    He committed to sharing a cleaned set of run files with hashes. The next step under this issue remains: obtain and preserve the originals privately, verify SHA-256 manifests and provenance, independently reproduce every reported count and accounting claim, and publish a sanitized derivation with clear caveats. Do not close this issue until those acceptance criteria are met. #16 has been merged as 5b42448; recovery is separate and still open.

  3. added 2 commits that reference this issue on Nov 16, 2026
  4. k2jac9 commented on Nov 16, 2026

    @k2jac9
    OwnerAuthor

    Evidence update — 16 November 2026 (AI-assisted, owner-authorized).

    PR #20 merged as e48a75a331b61cfa7f0b861805f485a0e5c47bc5. Its dated recovery verification checks the public archive from #19: 77 data-member hashes, 41,995 retained prompt/event/receipt bindings across 14 runs, recorded accounting, and 46 selected frozen-count comparisons. These checks pass; this establishes internal consistency, not provider authenticity or independently reconciled billing. All twelve runtime lanes passed merged-main CI.

    The machine-local executable paths in 11 published run manifests now have a separately identified, path-redacted successor proposed in draft PR #21, head 3e99b5de2e9e292bc75fdac3334290bffc7469bd. The original ZIP remains unchanged. The successor SHA-256 is f19127a49dea94e581e3714be3733ba480db7405632f1292b1f737273a23da04; 66 other data members remain byte-identical, and SANITIZATION.json maps all 80 prior members, including three superseded packaging metadata files. The mapping refers to the prior public archive; it does not claim a newly obtained private-original inventory. Existing public history still contains the removed paths.

    The publication report records scope and limits. Local validation passed 415 Python tests, five viewer tests, the full successor/source comparison, fleet checks, submission verification and runtime freshness. Independent code review found an extra-ZIP-bytes gap; failing regression tests demonstrated it before correction and the full suite passed again.

    Human handoff: fresh-checkout instructions and result template. Use the exact head SHA above. An AI rehearsal rebuilt the identical archive from a fresh local clone of the initial candidate; this is not the required second-human reproduction.

    Still open:

    • Source-holder private search/preservation inventory: original output directories/backups checked, timestamps, file sizes/hashes, preserved originals and missing artifacts. Keep machine-local paths and private inventory out of GitHub.
    • A second person's fresh-checkout reproduction with actual identity, source/input hashes, commands, exit codes and results.
    • Three stopped runs still lack completion manifests, decisions and databases. Their unrecorded activity/final liability remains unknown.
    • Provider billing/authenticity, historical signatures and other explicitly excluded statistical/audit checks remain unverified.

    Keep #17 open. The draft is a reviewable publication follow-up, not closure of the historical evidence gaps. No new paid model calls, experiment reruns or credentials were used.

    Hosted follow-up: all twelve runtime lanes passed for 3e99b5de2e9e292bc75fdac3334290bffc7469bd in both the push run and PR run. PR #21 remains a draft for the human evidence handoff; this does not satisfy the pending inventory or human reproduction.

  5. k2jac9 commented on Nov 16, 2026

    @k2jac9
    OwnerAuthor

    Status reconciliation — 16 November 2026 (AI-assisted, owner-authorized).

    The owner closed #17 at 10:03:31 UTC. That state is preserved; the preceding “keep open” and “draft PR” statements are superseded as repository-status descriptions. PR #21 is now merged at 206fc9997a7de5e7d46753f5e7f33d8fe222c486, and its merged-main CI passed.

    Closure does not supply new historical evidence. No additional private-original inventory or second-person reproduction receipt has been posted in this issue. The human verification handoff at merged main remains usable. The source-holder inventory, independent human reproduction, missing completion artifacts for three stopped runs, and provider authenticity/billing limitations remain as documented. The sanitized archive and automated consistency checks do not independently establish those facts.

    No acceptance boxes have been marked complete on the strength of closure alone. #18 remains open for the unreproduced historical TLS discrepancy; current-environment passes do not identify the cause in the unavailable environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions