Skip to content

Clarify experiment bookkeeping in program.md #306

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Gabrieliam42 wants to merge 1 commit into
base: master
Choose a base branch
from
+9 −2
Open

Clarify experiment bookkeeping in program.md #306

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions program.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ To set up a new experiment, work with the user to:
3. **Read the in-scope files**: The repo is small. Read these files for full context:
- `README.md` — repository context.
- `prepare.py` — fixed constants, data prep, tokenizer, dataloader, evaluation. Do not modify.
- `train.py` — the file you modify. Model architecture, optimizer, training loop.
- `train.py` — the source file you modify. Model architecture, optimizer, training loop.
4. **Verify data exists**: Check that `~/.cache/autoresearch/` contains data shards and a tokenizer. If not, tell the human to run `uv run prepare.py`.
5. **Initialize results.tsv**: Create `results.tsv` with just the header row. The baseline will be recorded after the first run.
6. **Confirm and go**: Confirm setup looks good.
Expand All @@ -23,7 +23,14 @@ Once you get confirmation, kick off the experimentation.
Each experiment runs on a single GPU. The training script runs for a **fixed time budget of 5 minutes** (wall clock training time, excluding startup/compilation). You launch it simply as: `uv run train.py`.

**What you CAN do:**
- Modify `train.py` — this is the only file you edit. Everything is fair game: model architecture, optimizer, hyperparameters, training loop, batch size, model size, etc.
- Modify `train.py` — this is the only source file you edit. Everything is fair game: model architecture, optimizer, hyperparameters, training loop, batch size, model size, etc.
- Do not modify `prepare.py` or any other source file.

Experiment artifacts and bookkeeping:
- You must append exactly one new row to `results.tsv` after every experiment, including `keep`, `discard`, and `crash`.
- You must append the row before any `git reset`, checkout, or revert.
- Updating `results.tsv` is required bookkeeping and is not a source-code modification.
- You may write or overwrite `run.log` for each run.

**What you CANNOT do:**
- Modify `prepare.py`. It is read-only. It contains the fixed evaluation, data loading, tokenizer, and training constants (time budget, sequence length, etc).
Expand Down