- Rust 1.97.1 (pinned in
rust-toolchain),just, Docker with buildx, and Harbor (uv tool install harbor). - Surfpool comes through Cargo, from the branch of surfpool#902 (cheatcode locks, SDK snapshots, clock restore), until it ships in a release.
- Nothing for the snapshot: its accounts are committed gzipped
(
snapshots/<name>/surfnet.json.gz, under 3 MB), andmtxdecompresses them into a temp-directory cache the first time a surfnet starts. To fork a fresh one, put a mainnet RPC URL in.envasMTX_FORK_RPC_URLand runjust snapshot-prepare snapshots/<name>, which writes the.gz.
just bots # build every bot's bin/bot
just grade bots/reference bench/tasks/phoenix-flash-crash/environment/practice/*.json
just test lint # unit tests and clippymtx grade runs exactly what the verifier runs, and bot logs land in
target/logs/. Don't run heavy builds while grading: a starved machine
slows the grade.
just bench # a batch, in jobs/bench/<batch>/
mtx bench review jobs/bench/<batch> # browse it (or `just review …`)
mtx bench publish jobs/bench/<batch> 0.9.0 # keep it, in bench/results/0.9.0/
mtx bench review 0.9.0 # browse a published release
mtx bench compare 0.9.0...0.8.0 [--open] # skill by task and model, and the change- Review page: a breadcrumb (
mtx › release › task › run › scenario), the tasks as a column of tiles, and an overview: a leaderboard of skill by model and task, and skill against cost. A task shows its equity curves and one grid of runs × scenarios. A run is one model's attempt (a Harbor trial); a scenario is one hidden market session every run shares. A row opens the run in a side panel: its numbers, then every scenario as a small PnL chart beside the picked one's timeline, the agent (time and token breakdowns, where its time went, context and cost over the run), the bot and the logs. A column replays that scenario for every run. Timelines need the detail files the verifier writes withmtx grade --details. - Published releases: small enough to commit, at a few hundred KB a
trial. A release holds the results, reports, session detail files, the
bot's source and the trajectory, gzipped. Publishing refuses any release
that would hold the batch's
.seedor an API key. The review page lets you switch between releases and compare them.
mtx scenario new "simulate a crash for SPCX" -a claude-code # writes scenario.json
mtx scenario play "simulate a crash for SPCX" -a claude-code # authors it, then plays it
mtx scenario play scenario.json # replays a saved onenew: a coding agent (claude-codeorcodex, over ACP) writes a Surfpool scenario. The agent works against a surfnet restored from the snapshot.mtxchecks what it writes and sends problems back until the scenario passes.play: serves a page with a chart per venue and dimension, and every override on a timeline. It has Play, Pause, Reset and Stop, plus a workbench with the surfnet's URLs and a live websocket stream.-vshows the logs instead of progress lines.
just ladder <family> # the skill ladder on fresh hidden sets
just image && just harbor-checks # oracle skill 1; nop, cheater, sneaky 0
just leak-audit # the agent's image holds no answer
harbor check bench/tasks/<task> # LLM review against Harbor's rubricAfter changing bots/reference, run just solution and recalibrate the
practice scenarios (just calibrate bots/reference FILES). Hidden sets need
no calibration, because the verifier grades the reference on them every
time. Record the change in the changelog.
Pick an issue labelled
new task,
and follow the trading-task-design
skill, which Claude Code loads in this repo. Write the task's README to
the template.
Bump tag in the Justfile, build the images and run the checks on them
(just image && just harbor-checks), commit, then:
just release # push both images (amd64 + arm64, ~40 min on Apple silicon),
# pin every task to its digest, lint, sync digests, publishCommit the pins and digests it writes. CI checks every push (fmt, clippy, tests, lint, calibrated scenarios), and once a week it builds the images and runs the oracle through Harbor.