feat(adapters): containerize 4 adapters for Terminal-Bench - #97
Open
Zlatanwic wants to merge 11 commits into
Open
feat(adapters): containerize 4 adapters for Terminal-Bench#97Zlatanwic wants to merge 11 commits into
Zlatanwic wants to merge 11 commits into
Conversation
The subprocess pi adapter used renderPiBaseUrlOverride, which writes only
the baseUrl override. pi's openai provider then defaults custom models to
openai-responses (POST {baseUrl}/responses); non-OpenAI backends
(DeepSeek, vLLM, any OpenAI proxy) don't implement Responses and return
404. Switch managed mode to renderPiModelRegistration, which registers the
model with api: 'openai-completions' so pi uses /chat/completions —
matching the headless library driver that already did this correctly.
Discovered running the pi adapter against DeepSeek via skvm run.
Long TB tasks (tb-circuit-fibsqrt, tb-feal-*-cryptanalysis) crashed with
RangeError: Out of memory. Root cause: piEventsToRunRecord received the
full PiEvent[] from parsePiNDJSON, which materialized every one of pi's
~30k emitted events (message_update / thinking / *_delta streaming
deltas make up ~99.9% of a 0.3-1.7 GB transcript). The builder only
consumes agent_end + message_end (~25 events); the other 30k parsed
objects sat in a retained array alongside the buffered stdout string,
driving peak heap to 10-32 GB.
Add piBuildRunRecordFromNDJSON: scan the stdout string with charCodeAt
(no split() array), pre-filter each line with .includes('"type":"agent_end"')
or '"type":"message_end"' to skip JSON.parse on 99.9% of lines, and
retain only the two event types the builder actually reads. Behaviorally
equivalent to the old path — a dedicated equivalence test compares
parsePiNDJSON+piEventsToRunRecord vs the new function on clean and
message_end-fallback transcripts.
Extracts piMessagesToRunRecord as a shared helper so streaming and
full-events paths cannot drift. piEventsToRunRecord retained for the
headless driver which already holds events in memory.
Tests: 29/29 pass in test/adapters/pi.test.ts (adds equivalence,
noise-skipping, and message_end-fallback cases). Verified against the
tb-circuit-fibsqrt log that previously OOM'd.
Follow-up to f8603a9: parser-side fix bounded RunRecord memory, but the
raw stdout string itself (0.3-1.7 GB for a long agentic transcript) was
still fully buffered by runSubprocess before pi.ts even saw it. Bench
resumed on f8603a9 crashed at RSS=12.16 GB / peak=18.61 GB with a Bun
segfault (SIGSEGV at a 2.29 TB virtual address, characteristic of a
very large string allocation) after 7 tasks. The Layer-1 fix cut
retention of parsed events; it did nothing about the raw string.
Layer 2 fix: give runSubprocess an optional stdoutSink option that
streams stdout bytes verbatim to a file lazily (no empty file if the
child produces nothing) with the same reader-cancellation semantics as
the existing string path. Pi adapter wires task.convLog.filePath in as
the sink — pi never populates convLog via logRequest/logResponse, so
finalize() early-returns on an empty entries[] and the streamed file
survives. Then piBuildRunRecordFromFile streams the same file back
line-by-line into the shared PiEventCollector (extracted from the two
existing paths so they cannot drift), retaining O(longest-line) bytes
instead of O(transcript).
Wall of writes:
before: pi -> stdout pipe -> stdout string (1 GB) -> parser
-> convLog file
after: pi -> stdout pipe -> convLog file
-> parser (stream reads back)
Peak heap projection: from 18.61 GB observed -> a few MB per task.
Also collapses the previous dual-write (buffer + convLog) into one.
Test files (test/adapters/pi.test.ts, test/core/subprocess.test.ts) are
scoring-adjacent under the PUA integrity policy and are intentionally
left uncommitted here for separate verifier ownership.
Add a container branch to the pi adapter so TB tasks (identified by
tbDockerImage on BenchTask) run inside skvm-pi-runtime with workDir
bind-mounted at /app. Model-issued shell commands (find /, grep -r, ...)
now execute in a clean Ubuntu root instead of the Windows Git-Bash host
via MSYS, fixing the tb-db-wal-recovery host-traversal timeout.
Components:
- src/core/docker-run.ts startContainer/execInContainer primitive,
symmetric to runSubprocess. Handles
MSYS_NO_PATHCONV, unique names, timeout kill,
idempotent cleanup.
- src/adapters/pi.ts runInContainer() method + branch on
task.tbDockerImage in run(). Existing host
path unchanged for non-TB tasks. Opt-out via
SKVM_PI_HOST_MODE=1.
- src/core/types.ts AgentAdapter.run() param type gains optional
tbDockerImage. Non-container adapters ignore.
- src/framework/runner.ts Thread BenchTask.tbDockerImage into adapter.run.
- docker/skvm-pi-runtime.Dockerfile
Ubuntu 24.04 + Node 20 + pi 0.67.68 pinned
to host node_modules version. CN apt mirror.
Verified: bench --adapter=pi --model=deepseek/deepseek-v4-pro
--tasks=tb-break-filter-js-from-html --conditions=no-skill → reward=1
in 4m19s. Replicates prior host-mode result on this task, confirming
the container branch is a functional equivalent.
Plan doc: plan/2026-07-02-containerized-pi-agent-for-tb.md
…er mode
Mirror skvm-pi-runtime for the two other adapters skvm will make
container-aware. Each preinstalls its own CLI on Ubuntu 24.04 + Node 20
with the same CN apt/npm mirror configuration proven by the pi image.
- skvm-claude-code-runtime npm i -g @anthropic-ai/claude-code
(built: 2.1.197, 1.39 GB)
- skvm-opencode-runtime npm i -g opencode-ai
(built: 1.17.13, 1.81 GB)
Unlike pi (which pins the version because the adapter writes an internal
models.json that the CLI parses), these CLIs are consumed purely through
documented command-line flags so version drift between host and container
is safe for A/B bench comparisons.
Adapter code that consumes these images comes in a follow-up commit.
Verify locally with:
docker run --rm skvm-claude-code-runtime:latest claude --version
docker run --rm skvm-opencode-runtime:latest opencode --version
Add a container branch to the claude-code adapter, mirroring the pi adapter (a2a283f). When a task carries tbDockerImage (and SKVM_CC_HOST_MODE is not set), claude runs inside skvm-claude-code-runtime with workDir bind-mounted at /app so model-issued shell commands (find /, grep -r, ...) execute in a Linux root instead of the Windows Git-Bash host via MSYS. - Existing host path unchanged; extracted into runOnHost() to keep the container branch isolated. - Container mode ships the managed-mode settings.json into a workDir-side .cc-sandbox and points CLAUDE_CONFIG_DIR at it via docker -e. Symmetric to pi's .pi-sandbox + PI_CODING_AGENT_DIR pattern. - Native mode inside container is explicitly rejected with a clear error (bench always uses managed; native support is a follow-up). - Opt-out for debugging: SKVM_CC_HOST_MODE=1. Verification status: TypeScript diagnostics pass; end-to-end reward=1 smoke NOT run because the current provider config has no Anthropic- protocol route (DeepSeek/LongCat are openai-compatible; Claude CLI needs /v1/messages). Verification will land when an Anthropic-compatible endpoint is available. Structural parity with the pi container branch (committed reward=1 on tb-break-filter-js-from-html) is the acting correctness signal for this change.
Add a container branch to the opencode adapter, mirroring the pi (a2a283f) and claude-code (fb06e89) adapters. When a task carries tbDockerImage (and SKVM_OC_HOST_MODE is not set), opencode runs inside skvm-opencode-runtime with workDir bind-mounted at /app so model-issued shell commands (find /, grep -r, ...) execute in a Linux root instead of the Windows Git-Bash host via MSYS. - Existing host path extracted into runOnHost() unchanged. - Container mode passes the cached OPENCODE_CONFIG_CONTENT JSON through the container's env — no sandbox file needed, since opencode reads provider config directly from that env variable. Simpler than pi and claude-code, both of which need a sandbox dir for their config files. - Native mode rejected inside container with a clear error; managed + openai-compatible only. Anthropic routes rejected because current setup path doesn't build a config for them anyway. - Opt-out for debugging: SKVM_OC_HOST_MODE=1. Verification: bench --adapter=opencode --model=deepseek/deepseek-v4-pro --tasks=tb-break-filter-js-from-html --conditions=no-skill exercised the full container path end-to-end in 1m12s / 9.6K tokens. Container lifecycle (start, exec, cleanup), workDir mount, OPENCODE_CONFIG_CONTENT plumbing, NDJSON parsing, and tb-grade verification all clean. tb-grade returned reward=0 (verifier ran normally; DeepSeek's XSS attempt didn't beat filter.py this run), which is the expected per-task-instance variability, not a code-path failure.
Add a container branch to the hermes adapter, mirroring the pi (a2a283f), claude-code (fb06e89), and opencode (73c074a) adapters. When a task carries tbDockerImage (and SKVM_HERMES_HOST_MODE is not set), hermes runs inside skvm-hermes-runtime with workDir bind-mounted at /app. - Existing host path extracted into runOnHost() unchanged; container branch is entirely additive. - Host CLI resolution (resolveHermesCmd) is deferred from setup() to runOnHost() lazily. Container-only bench runs no longer require hermes on the host — the previous eager resolveHermesCmd() would fail-fast even when the container branch owned the actual execution. This is a latent architectural gap in pi/claude-code/opencode too, hidden there because those CLIs happened to be reachable via host tiers. - Container mode runs TWO execs on the same running container: 1. hermes chat -q <prompt> -m <model> ... 2. hermes sessions export <sid> ... Both must land on the same container because session state lives in a SQLite DB under HERMES_HOME. The docker-run.ts container-per-task model (sleep 36000 PID 1) already supports multi-exec cleanly. - Managed config (config.yaml + .env, produced from providers.routes) is written to workDir/.hermes-sandbox and HERMES_HOME env points at the container-side path — same pattern as pi's .pi-sandbox and claude-code's .cc-sandbox. - Native mode rejected inside container with a clear error. - New Dockerfile skvm-hermes-runtime.Dockerfile: Ubuntu 24.04 + Python 3.12 + pip install hermes-agent==0.18.0 (PyPI, entry point `hermes`). Uses official archive.ubuntu.com because Docker Desktop vpnkit was hijacking aliyun to fake IPs on the build machine; the aliyun mirror rewrite from the older runtime Dockerfiles can be reintroduced when the split-DNS routing settles. Verification: bench --adapter=hermes --model=deepseek/deepseek-v4-pro --tasks=tb-break-filter-js-from-html --conditions=no-skill exercised the full container path end-to-end in 8m25s. Container lifecycle (start, chat exec, sessions export exec, cleanup), workDir mount, and tb-grade verification all clean; workDir produced out.html and the verifier ran normally. tb-grade returned reward=0 (verifier ran fine; DeepSeek's XSS attempt didn't beat filter.py this run), which is the expected per-task-instance variability, not a code-path failure. Known limitation: telemetry fell through to reduced-record path (Avg Tokens / Avg Cost "n/a") — the session_id trailer parsing works on the host branch but may need adjustment for container stdout buffering. Follow-up work; does not affect correctness of the workDir scored by tb-grade.
Hermes CLI 0.18.0 (and likely all versions since the initial adapter commit 64d1ea5) writes the `session_id: <id>` trailer to STDERR, not stdout — regardless of exit code. Empirically verified against skvm-hermes-runtime:latest with `hermes chat -Q --source=tool`: - STDOUT: tirith banner + agent's final reply - STDERR: blank line + `session_id: YYYYMMDD_HHMMSS_hexhash` The adapter was matching the regex against `stdout` in both the host branch (hermes.ts:438) and the container branch (hermes.ts:711), which silently never matched. Every hermes bench run degraded to the reduced-telemetry path: `hermes sessions export` was skipped (no sessionId to pass), Avg Tokens / Avg Cost rendered as "n/a" in the report, and per-token accounting was lost. This is a pre-existing bug from the initial commit, not a regression introduced by containerization. It surfaced during the container-mode work (commit e13b7a8) because the container branch's verification specifically tracked whether telemetry populated. Host branch has always been broken on machines that never ran hermes bench before. Fix: change both regex sources from `stdout.match(...)` to `stderr.match(...)`. Two-line change per branch; added explanatory comments with the empirical evidence so the next contributor doesn't "fix" it back to stdout. Verification: bench --adapter=hermes --model=deepseek/deepseek-v4-pro --tasks=tb-break-filter-js-from-html --conditions=no-skill Before: Avg Tokens n/a, Avg Cost n/a (session_id missing) After: Avg Tokens 28.1K, hermes sessions export ran successfully. The earlier "CRLF in Docker stdout" hypothesis (raised in the post-e13b7a8 review) was a false lead: hexdump of the failing run's convLog showed 0 CR bytes and 0 occurrences of `session_id` anywhere in the file. The trailer was never on stdout to begin with.
Replace the single-mirror sed rewrite (aliyun-only) in three runtime Dockerfiles with a dual-source approach: keep the official archive.ubuntu.com / security.ubuntu.com entries in ubuntu.sources untouched, and append aliyun as a second DEB822 stanza. apt aggregates both stanzas and pulls from whichever responds. When aliyun is reachable (the common case in CN), builds use the faster mirror. When aliyun is hijacked by Docker Desktop vpnkit to fake IPs (198.18.0.x — the failure that hit skvm-hermes-runtime.Dockerfile mid-build on 2026-07-05), apt transparently falls back to the official archive and the build still succeeds. Hardens: - docker/skvm-pi-runtime.Dockerfile - docker/skvm-claude-code-runtime.Dockerfile - docker/skvm-opencode-runtime.Dockerfile skvm-hermes-runtime.Dockerfile already uses official-only (it was the incident that surfaced the issue). It is intentionally left as-is — its comment documents the incident for future contributors. A follow-up could unify all four Dockerfiles on the dual-source pattern. Why DEB822 multi-stanza (not a separate .list file): apt treats the same suite appearing in two different source files as "configured multiple times" and emits a warning on every apt-get update. Putting aliyun as a second stanza in the same ubuntu.sources file is treated as "the same repository mirrored at two URIs" and produces no warning. Verification (per image): docker build -t skvm-<name>-runtime:latest -f docker/skvm-<name>-runtime.Dockerfile . docker run --rm skvm-<name>-runtime:latest <cli> --version docker run --rm skvm-<name>-runtime:latest grep -c mirrors.aliyun.com /etc/apt/sources.list.d/ubuntu.sources All three rebuilt successfully (pi 1.28GB, claude-code 1.39GB, opencode 1.81GB) with the aliyun stanza present and apt-get update observed pulling from both mirrors in parallel.
7 tasks
There was a problem hiding this comment.
Pull request overview
Adds container-mode execution paths to multiple CLI-based adapters so Terminal-Bench tasks can run in a predictable Linux filesystem environment (workDir bind-mounted at /app). This is complemented by new subprocess stdout streaming and pi NDJSON streaming parsing to keep memory bounded for very large agent transcripts.
Changes:
- Introduce a shared Docker container lifecycle helper (
startContainer/execInContainer) and wire it into pi/claude-code/opencode/hermes adapters whentbDockerImageis present. - Add
stdoutSinksupport torunSubprocess(and tests) to stream large stdout payloads directly to disk instead of buffering in-memory. - Add streaming pi transcript parsing helpers (
piBuildRunRecordFromNDJSON/piBuildRunRecordFromFile) plus unit tests.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| test/core/subprocess.test.ts | Adds coverage for stdoutSink streaming semantics (large output, timeout, lazy file creation). |
| test/adapters/pi.test.ts | Adds tests ensuring streaming pi parsing stays behaviorally equivalent to the old in-memory path. |
| src/framework/runner.ts | Threads tbDockerImage into adapter run() context via a safe cast for TB-imported tasks. |
| src/core/types.ts | Documents new tbDockerImage run parameter on the adapter interface. |
| src/core/subprocess.ts | Implements stdoutSink to stream stdout to a file and returns stdoutFile metadata. |
| src/core/pi-runtime.ts | Refactors run-record building and adds streaming/lower-memory pi NDJSON parsing paths. |
| src/core/docker-run.ts | New helper for starting a long-lived container and executing commands within it (with timeout handling). |
| src/adapters/pi.ts | Adds container-mode branch; switches host-mode to stream stdout to convLog when available and parse from file. |
| src/adapters/opencode.ts | Adds container-mode branch; passes cached route env/config into container and reuses shared docker helper. |
| src/adapters/hermes.ts | Adds container-mode branch; fixes session_id parsing to read trailer from stderr; supports two execs per run. |
| src/adapters/claude-code.ts | Adds container-mode branch; uses cached route env and managed settings written into a workDir-mounted sandbox. |
| docker/skvm-pi-runtime.Dockerfile | Adds runtime image for pi CLI execution inside Ubuntu container. |
| docker/skvm-opencode-runtime.Dockerfile | Adds runtime image for opencode CLI execution inside Ubuntu container. |
| docker/skvm-hermes-runtime.Dockerfile | Adds runtime image for hermes CLI execution inside Ubuntu container. |
| docker/skvm-claude-code-runtime.Dockerfile | Adds runtime image for claude-code CLI execution inside Ubuntu container. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary Adds container-mode support to 4 adapters (pi, claude-code, opencode, hermes) for running Terminal-Bench tasks. Each adapter now has two branches: host mode (agent runs on Windows/macOS/Linux host) and container mode (agent runs inside a Docker container with the workDir bind-mounted at
/app). This PR builds on #94 (pi-adapter-streaming) — the streaming infrastructure and openai-completions fix are prerequisites. ## Why Containerization Terminal-Bench tasks are designed for Linux containers. Some tasks (e.g.tb-break-filter-js-from-html) include shell commands likefind /that traverse the entire filesystem — on a Windows host under Git Bash, this times out or hits permission errors. Running the agent inside a clean Ubuntu container gives it a predictable Linux root to walk. ## Architecture Host agent + container verifier (validated in path-C pilot): 1. workDir seeding (PR C): Before the agent starts,docker run --rmcopies the TB image's/appcontents to the host workDir 2. Agent runs on host OR in container (this PR): Adapters checktask.tbDockerImage— if set andSKVM_PI_HOST_MODEis unset, launch a container with workDir mounted at/app3. Verifier runs in TB image (PR C):tb-gradeevaluator mounts the same workDir back into the TB image at/appand runstests/test.sh## Per-Adapter Changes ### 1. Pi adapter (src/adapters/pi.ts) - New method:runInContainer()— launchesskvm-pi-runtime:latest, bind-mounts workDir at/app, runsdocker exec pi ...withPI_CODING_AGENT_DIRinside the mount - Managed mode only: Native mode (user's~/.pi/agent/) would need extra bind-mount work; bench uses managed mode exclusively - models.json reuse: The route override JSON (this.modelsJson) is written inside workDir so it auto-mounts with the volume ### 2. Claude-code adapter (src/adapters/claude-code.ts) - New method:runInContainer()— launchesskvm-claude-code-runtime:latest, bind-mounts workDir at/app- Managed mode only: Same rationale as pi ### 3. OpenCode adapter (src/adapters/opencode.ts) - New method:runInContainer()— launchesskvm-opencode-runtime:latest, bind-mounts workDir at/app- Managed mode only: Same rationale as pi ### 4. Hermes adapter (src/adapters/hermes.ts) - New method:runInContainer()— launchesskvm-hermes-runtime:latest, bind-mounts workDir at/app- Telemetry fix: Readsession_idtrailer from stderr, not stdout (pi's NDJSON goes to stdout, session_id to stderr) ## Shared Container Utility (src/core/docker-run.ts) New module with two helpers: -startContainer()—docker run -dwith workDir bind-mount, returns{ name, cleanup }-execInContainer()—docker execwith stdout streaming (reusesrunSubprocess+stdoutSinkfrom PR #94) ## Docker Runtime Images 4 new Dockerfiles indocker/: -skvm-pi-runtime.Dockerfile— Ubuntu 24.04 + Node.js 20 +@mariozechner/pi-coding-agent@0.67.68via npm -skvm-claude-code-runtime.Dockerfile— Ubuntu 24.04 + Node.js 20 +@anthropic-ai/claude-codevia npm -skvm-opencode-runtime.Dockerfile— Ubuntu 24.04 + Node.js 20 +opencode-aivia npm -skvm-hermes-runtime.Dockerfile— Ubuntu 24.04 + Python 3.12 +hermes-agent==0.18.0via pip Apt source hardening: The pi, claude-code, and opencode images use official Ubuntu plus an Aliyun mirror in DEB822 multi-stanza format. The hermes image deliberately keeps official Ubuntu sources because they are reachable in the environment where the Aliyun route is intercepted. ## Type Updates -BenchTask.tbDockerImage(added in PR C) signals the adapter to use container mode -RunnerContext.tbDockerImage(new) threads the image name through to adapters ## Dependency Note This PR is stacked on #94 (pi-adapter-streaming). GitHub shows only the 7 containerization commits as the diff. Upstream should merge #94 first, then this PR. ## Test plan - [x]bunx tsc --noEmitpasses - [x] Built all 4 runtime images successfully - [x] Smoke test: pi adapter + TB tasktb-break-filter-js-from-htmlin container mode → reward=1 - [x] Verified hermes telemetry fix (session_id from stderr) - [x] Verified dual-source apt fallback in containers ## Scope Additive container branches. Host mode unchanged. No breaking changes. NewrunInContainer()methods, newdocker-run.tsmodule, 4 new Dockerfiles.