Auditable Auto Research

A small, domain-extensible reference implementation of auditable Auto Research. It turns code-agent trial-and-error into a controlled campaign:

1. expose one research axis at a time;
2. score candidates only with inner cross-validation;
3. keep an append-only ledger and non-overwritten, hashed trial workspaces;
4. freeze the selected source tree;
5. reveal the outer holdout only through a separate certification command.

The generic core owns orchestration, isolation, selection, lineage, and certification. Domain knowledge lives in adapters. A numeric-materials adapter is included as the first working example; it can be replaced without changing the core. The repository contains no redistributed benchmark dataset, model checkpoint, private endpoint, or credential. A compact, claim-bearing materials paper archive is released separately under paper_artifact/.

Repository layout

src/autoresearch_core/       task-agnostic runner, ledger, workspace and backends
src/autoresearch_adapters/   replaceable domain adapters
  materials/                 numeric-materials worker and research-axis definitions
  materials/seed/            minimal editable candidate pipeline
configs/                     credential-free example campaign
docs/METHOD.md               experimental contract
docs/ADAPTER_GUIDE.md        how to plug in formulas, structures or another domain
RUNBOOK.md                   installation, launch, resume and certification
tests/                       offline unit and end-to-end tests
paper_artifact/              minimal public data and code for the materials paper

Materials paper artifact

The compact archive under paper_artifact/ contains all 701 retained trial results, the seven frozen winners, certification scores, the exact small candidate data files and embedding tables used by selected interventions, and standard-library scripts that verify the paper's headline claims and file hashes. Matbench data are fetched through the upstream package and are not redistributed.

cd paper_artifact
python scripts/verify_manifest.py
python scripts/verify_claims.py

The protocol in one picture

external data ──> outer fold 0 ─────────────────────────────> sealed holdout
       │
       └────────> outer-train ──> inner 5-fold score ──> Agent search
                                                    │
feature OR representation OR model OR data files <─┘
                                                    │
                                      frozen best source tree
                                                    │
                                      explicit `certify` command

The run command never calls the holdout path. Development and holdout evaluation both execute candidate code in a fresh Python subprocess, preventing module-state carry-over between fitting and evaluation.

Quick start

Requirements: Python 3.11+ and, for live research, a code-agent CLI that accepts a prompt on stdin and edits its current working directory.

python -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev]'

cp .env.sample .env
cp configs/campaign.example.toml campaign.toml

# Create synthetic data only for the local smoke test; no data are committed.
python scripts/make_smoke_data.py --output /tmp/material_smoke.npz
# Set evaluator.data_path in campaign.toml to /tmp/material_smoke.npz.

autoresearch smoke campaign.toml
pytest -q

smoke always uses the offline mock backend and writes to a separate _smoke run directory. It makes no API call. Inspect it with autoresearch status --smoke campaign.toml.

For a live campaign, authenticate the chosen Agent CLI, set backend="command" in campaign.toml, and run:

autoresearch run campaign.toml
autoresearch status campaign.toml

After the campaign and analysis plan are frozen:

autoresearch certify campaign.toml

See RUNBOOK.md before using a scientific dataset.

Agent backend and credentials

The core does not implement a provider API client. It runs the configured command, sends the research prompt on stdin, and expects the command to edit the staged workspace. This works with Claude Code or another compatible coding-agent CLI.

Copy .env.sample to the repository-root .env. The command backend loads only that file and injects its variables into the Agent CLI subprocess. Exported process variables take precedence. The real .env is ignored by Git, is never copied into a trial, and is never written to a manifest or log.

Users whose CLI already manages authentication may leave the sample variables blank.

Included materials adapter

The included evaluator expects an external .npz file:

np.savez_compressed("/outside/repo/task.npz", X=X_numeric, y=y_numeric)

• X: finite numeric array shaped [n_samples, n_features]
• y: finite numeric array shaped [n_samples]
• regression metrics: mae, rmse
• classification metrics: accuracy, roc_auc

The path stays in local campaign.toml; both are ignored or kept outside Git. Formula, crystal-structure, graph, and multi-endpoint projects can extend or replace this adapter as described in docs/ADAPTER_GUIDE.md.

Add another domain adapter

The core depends on three adapter-owned definitions: a seed source tree, an evaluator worker module, and the research-axis edit surfaces. To add a domain:

1. create src/autoresearch_adapters/<domain>/seed/ with a stable candidate API;
2. implement a fresh-process worker that loads data, applies leakage-safe splits, imports the staged candidate, and prints the standard JSON score payload;
3. define the domain's research axes as editable file patterns and instructions;
4. point campaign.seed_dir and evaluator.worker_module at the new adapter;
5. keep data external and add adapter-specific tests before a live campaign.

Formula/structure parsing, scientific metrics, grouped or temporal splits, and data provenance therefore remain domain-owned. Resume, ledgers, file boundaries, Agent CLI execution, source hashing, inner-CV selection, and explicit holdout certification remain core-owned.

Axis isolation

For paper-grade attribution, create a separate config and run_dir for every axis:

runs/task_feature/
runs/task_representation/
runs/task_model/
runs/task_data/

Enable exactly one specialist in each config. The runner enforces this by default. Its editable file is the only source surface the agent may change. Joint search can be enabled explicitly with axis_isolated=false, but its result has a different estimand and should not be mixed with isolated-axis claims.

What to cite and report

At minimum report the dataset/split digest, outer and inner fold definitions, seed source hash, agent command/model, editable surface, all trial statuses, development selection rule, frozen source hash, and holdout certification. The run directory contains these fields in manifest.json, ledger.jsonl, best.json, hashed trial workspaces, and holdout_certification.json.

License

Apache License 2.0. Dataset and model licenses remain the responsibility of each adapter user.