feat(bench): Terminal-Bench 2.1 harness integration - #96
Open
Zlatanwic wants to merge 5 commits into
Open
Conversation
Import Terminal-Bench 2.1 tasks into skvm-data/tasks via `skvm bench --import=terminalbench --path=<tb-repo>`. Converts task.toml + instruction.md into a skvm task.json: instruction -> prompt (with /app/ rewritten to ./ and a guardrail telling the agent not to run the container-only verifier), docker_image -> tbDockerImage + evaluator payload, tests/ copied with CRLF->LF normalisation. Extends BenchTaskFileSchema with optional tbDockerImage / tbTestsDir so TB metadata is visible at the task level; the tb-grade evaluator (next commit) reads the same data from the custom criterion payload since CustomEvalContext has no task object. Loader and skvm run path transparently pass the new fields through.
Custom evaluator that runs a TB task's tests/test.sh INSIDE the task's docker image and reads /logs/verifier/reward.txt (0/1) -> score 0.0/1.0. One container per run, created and torn down within a single run() call (finally block) so errored tasks don't leak containers. Architecture (validated by path-C pilot): the agent ran on the host against workDir; this evaluator mounts that same workDir back into the image at /app and the LF-normalised tests/ at /tests:ro, then docker exec bash /tests/test.sh. Reads dockerImage/testsDir/verifierTimeoutSec from criterion.payload (written by the terminalbench importer). Pre-flight docker daemon probe + MSYS_NO_PATHCONV for Git Bash path safety.
For Terminal-Bench tasks (tbDockerImage set), copy the image's /app contents into the host workDir before the agent runs, so the host-side agent can read the task's actual files (e.g. filter.py). The tb-grade evaluator later mounts this same workDir back into the image at /app, so the agent's edits land in the container faithfully. No-op for non-TB tasks. Without this, the agent runs in an empty workDir, can't find the task files referenced in the prompt, and spins until OOM. Uses a throwaway docker run --rm with MSYS_NO_PATHCONV for Git Bash path safety.
Add path-specific ignore rules resolved during S641/S642 (2026-07-05):
- /terminal-bench-2-1/ external Terminal-Bench 2.1 checkout (its own
git repo, ~3 GB dataset). Existing untracked line
kept for compatibility; this one is redundant but
harmless and self-documents intent.
- /tb2.1-skills/ skill files imported from Terminal-Bench 2.1
into the local dataset area. SkVM's TB glue code
(src/bench/importers/terminalbench.ts,
src/bench/evaluators/tb-grade.ts, related tests
and docs) IS tracked; only the imported skill
content is not.
- /bench-results/ ephemeral bench run outputs (per-session logs,
report.md/json). SkVM writes these under
$SKVM_CACHE by default, but a stale symlink or
local run can leak them here.
- /benchmark-results/ same as above with the alternative name some
scripts use.
- .playwright-mcp/ per-session Playwright MCP snapshots and traces.
- plan/**/*.pdf PDF exports of plan docs (regenerated from md).
- plan/**/*_tmp.html intermediate render output from plan tooling.
Companion commit removes tb2.1-skills from the index so these rules take
effect (gitignore alone doesn't untrack already-tracked files).
7 tasks
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds first-class support for importing and evaluating Terminal-Bench 2.1 tasks within the existing bench framework.
Changes:
- Extend bench task schema/loader to carry Terminal-Bench metadata (
tbDockerImage,tbTestsDir) - Add a Terminal-Bench importer (
bench --import=terminalbench) with optional task filtering - Add a
tb-gradecustom evaluator that runs TB’s verifier inside the task’s Docker image, and seed hostworkDirfrom/app
Reviewed changes
Copilot reviewed 8 out of 9 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| src/run/index.ts | Propagates TB-specific task metadata into loaded run tasks |
| src/bench/types.ts | Adds TB metadata fields + schema support for task.json |
| src/bench/loader.ts | Persists TB metadata when reading/writing tasks |
| src/bench/index.ts | Adds terminalbench import mode + --tasks filtering |
| src/bench/importers/terminalbench.ts | New importer converting TB 2.1 repo tasks into SkVM bench tasks |
| src/bench/evaluators/tb-grade.ts | New custom evaluator that runs TB verifier in Docker and reads reward.txt |
| src/bench/evaluators/index.ts | Registers the new tb-grade evaluator |
| src/bench/conditions/run-condition.ts | Seeds host workDir with image /app contents for TB tasks |
| .gitignore | Ignores local TB repo clones and generated artifacts |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds SkVM harness support for Terminal-Bench 2.1 tasks — importer, container-based evaluator, and workDir seeding so host-side agents can read task files. This is the infrastructure half of the TB integration validated in the path-C pilot.
1. Terminal-Bench importer (
src/bench/importers/terminalbench.ts)Imports TB 2.1 tasks from the external
terminal-bench-2-1repo:task.toml(docker_image, pass_criterion),instruction.md(the agent prompt), andtests/(the verifier script)/apppaths in instructions to relative (for host-side agent compatibility)tbDockerImage,pass_criterion) in the BenchTask schemaskvm-data/tasks/tb-*/withhostReady: false(agent runs on host, but the task's verifier needs Docker)CLI:
skvm bench --mode=import --source=terminalbench --path=<tb-repo>2. tb-grade evaluator (
src/bench/evaluators/tb-grade.ts)Custom evaluator that runs the TB verifier (tests/test.sh) inside the task's Docker image and reads the reward.txt signal (0 or 1):
/app/tests:robash /tests/test.shinside the container/logs/verifier/reward.txt(0=fail, 1=pass)finallyto prevent leaksThis is the container-verifier half of the "host agent + bind-mount" architecture.
3. workDir seeding (
src/bench/conditions/run-condition.ts)For TB tasks, seed the host workDir with the image's
/appcontents before the agent starts:docker run --rm -v <workDir>:/out <image> sh -c "cp -r /app/. /out/"to copy the task's files out/app/filter.pybecomes<workDir>/filter.py)4. gitignore rules
Narrow TB-related gitignore entries:
/terminal-bench-2-1/— the cloned TB repo (3rd-party, large)/tb2.1-skills/— imported skills dataset (not source code)/bench-results/,/benchmark-results/— generated run outputssrc/bench/importers/, evaluators/, CLI, tests, docs)Why it's standalone
This PR is infrastructure only — the importer, evaluator, and workDir seeding. The adapter container branches (pi, claude-code, opencode, hermes) are a separate PR (D) because they touch adapter/runtime internals. This PR gives you the TB harness; PR D gives you containerized agents to run in it.
Test plan
bunx tsc --noEmitpasses/appcontents to the host before the agent startsScope
Additive infrastructure. No breaking changes. New evaluator type (
tb-grade), new importer (terminalbench), new task schema fields (tbDockerImage,pass_criterion).