Nanocodex VM

Retained libkrun workspaces and canonical workspace tools for Nanocodex.

nanocodex-vm is an experimental, unpublished, library-first crate. An application owns one [VmWorkspace] for each isolation boundary, gives its [tools::VmTools] to one or more agents, and keeps the VM alive across sequential turns. The crate does not own agent scheduling, evaluation policy, payment providers, or secrets.

The normal API is:

[VmWorkspaceBuilder] materializes and launches a retained private workspace;
[image] prepares immutable root images;
[tools] stages the companion guest and exposes VM-backed Nanocodex tools; and
[host] contains the lower-level libkrun, launch-record, networking, and egress types used by specialized applications.

The crate root intentionally re-exports only [VmWorkspace], [VmWorkspaceBuilder], and [VmWorkspaceError].

Use VM-backed workspace tools

Build the static Linux companion with just build-vm-guest, prepare one read-only runtime disk, and launch a private copy of an immutable root:

use nanocodex_vm::{
    image::{CachePolicy, VmImageBuilder},
    tools::GuestRuntimeDisk,
};

# async fn prepare() -> Result<(), Box<dyn std::error::Error>> {
let runtime = GuestRuntimeDisk::prepare(
    "target/aarch64-unknown-linux-musl/debug/nanocodex-vm-guest",
    ".cache/nanocodex/vm",
)?;
let image = VmImageBuilder::new("nanocodex", runtime.path())
    .vmm_args(["vm-run-config", "--config"])
    .firmware_directory(".cache/libkrunfw/libkrunfw")
    .prepare(
        "tasks/project/environment",
        10 * 1024 * 1024 * 1024,
        ".cache/nanocodex/vm",
        CachePolicy::Reuse,
    )
    .await?;
let workspace = image.private_workspace(
    ".nanocodex/sessions/018f/root.ext4",
    "nanocodex",
)?
.vmm_argument("vm-run-config")
.vmm_argument("--config")
.guest_runtime_disk(runtime.path())
.firmware_directory(".cache/libkrunfw/libkrunfw")
.launch()
.await?;

let tools = workspace.tools_builder().build()?;
// Pass `tools` to `Nanocodex::builder(...).tools(tools)`.

drop(tools);
workspace.shutdown().await?;
# Ok(())
# }

High-fanout ephemeral attempts use guest OverlayFS instead of copying that retained workspace shape. [host::VmConfig::overlay_ext4] boots the runtime disk read-only, mounts the prepared task disk read-only as the lower layer, and sends all mutations to a fresh sparse ext4 upper created by [host::create_sparse_overlay_disk]. Reset is deletion of that upper disk; the host filesystem needs ordinary sparse-file support, not reflinks, XFS, or a host OverlayFS mount. Attempts configured for rootfs retention continue to use standalone private ext4 copies so retained artifacts remain self-contained.

use nanocodex_vm::{
    host::{
        EgressLease, Network, VmConfig, create_sparse_overlay_disk,
        overlay_guest_command,
    },
    tools::VmToolSession,
};
use tokio::process::Command;

# async fn launch() -> Result<(), Box<dyn std::error::Error>> {
let upper = ".nanocodex/attempts/018f/upper.ext4";
create_sparse_overlay_disk(upper, 10 * 1024 * 1024 * 1024)?;
let config = VmConfig::overlay_ext4(
    ".cache/nanocodex/vm/runtime.ext4",
    ".cache/nanocodex/vm/prepared-task.ext4",
    upper,
)
.cpus(2)
.memory_mib(1024)
.network(Network::Disabled);
let session = VmToolSession::spawn_configured(
    Command::new("dedicated-vmm-process"),
    config,
    overlay_guest_command("/workspace", ""),
    EgressLease::disabled(),
)
.await?;
session.shutdown().await?;
# std::fs::remove_file(upper)?;
# Ok(())
# }

The caller owns upper-disk retention and deletion. Drop all session/tool capabilities and complete [tools::VmToolSession::shutdown] before removing the disk. Overlay startup creates only the requested workspace; harness- or application-specific directories remain the caller's responsibility.

[VmWorkspace::tools] returns a clone-cheap capability suitable for NanocodexBuilder::tools_factory. Every clone routes to the same retained guest runtime, filesystem, and interactive shell sessions. The non-cloneable workspace owner is the graceful-shutdown capability; drop agents, registries, and cloned tool handles before calling [VmWorkspace::shutdown].

The default tool selection keeps web search, image generation, and update_plan on the host. It replaces only exec_command, write_stdin, apply_patch, and view_image, preserving their standard model-visible names and schemas.

Session control and cleanup

Specialized applications that construct a lower-level [tools::VmToolSession] can run trusted setup and harness commands with [tools::VmCommand]. Commands have explicit time and combined-output bounds. Dropping an in-flight command request queues cancellation; the guest terminates the command's process group on cancellation, timeout, output overflow, or session shutdown.

[tools::VmCommand::mirror_output] additionally truncates two harness-owned guest files before launch and updates them as stdout and stderr arrive. This is intended for observing a long-running command from another request. It does not relax the command's retained-output bound or change its terminal result.

At an agent-lifecycle boundary, [tools::VmToolSession::terminate_tool_processes] cancels processes and interactive shells owned by the workspace-tool runtime while leaving the VM, filesystem, and host-control channel alive. It does not claim to kill a process that deliberately detached from the runtime's managed process group. Call [tools::VmToolSession::memory_observation] for best-effort peak host RSS, guest memory use, and guest OOM evidence. Missing telemetry is represented by absent fields so it cannot replace the command or agent failure being diagnosed.

Host, VMM, and guest ownership

The retained path has three processes:

embedding application
  ├─ owns VmWorkspace, agent state, tools, policy, and egress leases
  └─ spawns a dedicated VMM process from a mode-0600 launch record
       └─ libkrun starts one Linux guest
            └─ nanocodex-vm-guest serves workspace tools over the console

The application process does not call libkrun after starting an async runtime. Instead, [host::VmProcessConfig::write_private] writes a complete private launch record and the dedicated VMM entry point calls [host::VmProcessConfig::run] synchronously. This process boundary also keeps the macOS hypervisor entitlement on the smallest executable and prevents guest environment values from appearing in command-line arguments.

The shipped nanocodex binary provides that entry point as the hidden vm-run-config command. A library consumer may provide the same small entry point in its own executable:

use nanocodex_vm::host::VmProcessConfig;

# fn vmm(config_path: std::path::PathBuf) -> Result<(), Box<dyn std::error::Error>> {
VmProcessConfig::read(config_path)?.run()?;
# Ok(())
# }

On macOS, the VMM executable must be signed with the com.apple.security.hypervisor entitlement. Ad-hoc signing is sufficient for local development; distribution uses the application's normal signing identity. just build-eval-host uses nanocodex-vm.entitlements to produce the prepared repository build; runtime execution only validates the entitlement.

Linux uses the same Rust API and needs no code signing. Running VMs requires /dev/kvm and libkrunfw.so.5. The supported x86_64 static guest build uses a musl 1.2.3 ABI floor because the pinned libkrun KVM path requires statx.

Root and runtime disks

[image::VmImageBuilder] resolves a constrained Dockerfile/OCI build context into a content-addressed immutable ext4 root. The final OCI/Dockerfile working directory, process environment, and detected shell are retained with [image::PreparedRootDisk]. Its [image::PreparedRootDisk::private_workspace] method is the normal bridge to the retained workspace API: it makes a no-clobber reflink or sparse copy and applies that runtime metadata. Writable roots are session-private.

Build cache identity includes the Dockerfile and deterministic context, base manifest digests, architecture and disk size, VMM arguments and exact VMM/guest-runtime/configured-firmware file identities, CPU/memory/address-family policy, host resolver configuration, network mode, and the egress lease's non-secret cache scope. Set the firmware directory explicitly when firmware upgrades must invalidate cache entries; omitted system firmware is caller-managed stable runtime state. Cached OCI blobs are SHA-256 checked before their metadata fast path is established. Cached blobs and ext4 disks are published atomically and made read-only; changes to their inode, size, modification/change time, or permissions force validation or rebuilding. The caller-selected cache directory remains trusted application state rather than a security boundary against the same OS user.

By default, the complete VMM executable is part of build-cache identity. An application whose small VMM entry point is embedded in a frequently changing binary may set [image::VmImageBuilder::vmm_build_cache_identity] to a stable, non-secret semantic version. This is an explicit correctness promise: the caller must change it whenever the VMM's Dockerfile-build behavior changes. The remaining runtime, firmware, resource, network, resolver, and egress inputs are still hashed independently. Empty and excessively large identities are rejected.

Prepared roots retain the configured UID-zero account's supported bash or sh shell when that executable exists, then fall back to probing conventional shell paths. A cache hit revalidates the shell from the immutable disk instead of trusting metadata written by an older release.

Dockerfile build VMs temporarily install the current usable host resolver and restore the image's original /etc/resolv.conf before a stage disk can be published. Retained private ext4 workspaces install resolver configuration at boot instead, so immutable images never retain host-specific DNS. Offline and gvproxy workspaces do not receive host resolver injection. Directory roots are host-backed development escape hatches and are not rewritten.

[tools::GuestRuntimeDisk::prepare] hashes the exact companion ELF and atomically publishes a reusable 128 MiB ext4 disk. The runtime disk is mounted read-only, independently from the writable project root. That keeps the guest implementation identical across a sweep without mutating every root image.

Directory roots are a lower-level development escape hatch. They must already contain /usr/local/bin/nanocodex-vm-guest, and direct virtiofs access does not provide the same host mount-namespace isolation as a private ext4 root.

Host/guest RPC protocol

This section is the complete current wire contract implemented by the host session and nanocodex-vm-guest. The protocol is private implementation detail: host and guest artifacts are built from the same Nanocodex revision, and there is currently no version negotiation or cross-version compatibility promise. Applications use the typed Rust API rather than constructing frames.

Transport and envelope

The dedicated VMM's standard streams carry the guest's default virtio console:

host to guest: VMM stdin;
guest to host: VMM stdout; and
diagnostics only: VMM stderr.

stdin and stdout are newline-delimited JSON. Each frame is one UTF-8 JSON object followed by \n; readers also accept \r\n. The newline is not part of the 64 MiB frame limit. Binary fields use standard padded base64. There is no authentication, checksum, compression, streaming sub-frame, or handshake beyond ready, because the transport is a private pipe to the owned VMM process.

Every frame has the externally tagged envelope:

{"kind":"ready","payload":{"id":0}}

kind is snake case. Every request carries a host-assigned u64 id, and exactly one response carries the same ID unless the request is cancelled or the session fails. Responses may arrive in any order. The host allows at most 63 ordinary requests to await responses; the guest executes at most 64 requests concurrently, leaving capacity for control traffic.

Each ordinary request emits a vm.tool.rpc span. Its rpc.admission.duration_ns field measures time waiting for one of those 63 host slots, while rpc.queue.duration_ns measures the later wait to enter the bounded writer channel. duration_ns covers the complete RPC.

Requests and responses

ready establishes that the guest runtime is accepting work:

{"kind":"ready","payload":{"id":0}}
{"kind":"ready","payload":{"id":0,"error":null}}

tool executes one canonical workspace tool:

{"kind":"tool","payload":{"id":1,"tool":"exec_command","input":{"function":{"arguments":{"cmd":"pwd"}}},"context":{"model":"gpt-5.6","session_id":"session-1","call_id":"call-1","output_token_budget":10000}}}
{"kind":"tool","payload":{"id":1,"execution":{"output":"/app\n","success":true,"structured_result":null,"metadata":null,"process_trace":{"exit_code":0,"session_id":null,"original_token_count":null,"output_bytes":5,"wall_time_seconds":0.01}},"error":null}}

The normal adapter sends exec_command, write_stdin, apply_patch, or view_image. input is exactly one of:

{"function":{"arguments":{"cmd":"pwd"}}}
{"freeform":{"input":"*** Begin Patch\n...\n*** End Patch\n"}}

Function arguments remain opaque JSON. context contains model, session_id, call_id, and output_token_budget; conversation history is not copied into the guest context. execution.output is either a string or the canonical ordered multimodal array of input_text, input_image, and input_audio objects. structured_result and metadata are opaque JSON or null. process_trace is null or contains exit_code, session_id, original_token_count, output_bytes, and wall_time_seconds.

An execution with "success":false is a model-visible tool failure. A failure of the RPC/tool runtime itself instead uses "execution":null and a non-null "error". Exactly one of execution and error is present.

The remaining control methods have these payloads:

`kind`	Request payload after `id`	Response payload after `id`
`write_file`	`path`, base64 `contents`, Unix `mode`, optional `modified_unix_seconds`	`error`
`create_directory`	`path`, Unix `mode`, optional `modified_unix_seconds`	`error`
`read_file`	`path`	base64 `contents` or `error`
`memory`	none	optional `total_kib`, optional `minimum_available_kib`, `oom_kills`, `error`
`execute`	`program`, `arguments`, `current_directory`, `environment`, `timeout_millis`, `max_output_bytes`, optional `stdout_mirror`, optional `stderr_mirror`	`exit_code`, base64 `stdout`, base64 `stderr`, `error`, `timed_out`, `output_limit_exceeded`
`cancel`	`target_id`	`error`
`terminate_tool_processes`	none	`error`
`shutdown`	none	`error`

Concrete examples:

{"kind":"write_file","payload":{"id":2,"path":"/tmp/input","contents":"aGVsbG8K","mode":420}}
{"kind":"write_file","payload":{"id":2,"error":null}}
{"kind":"create_directory","payload":{"id":3,"path":"/tmp/results","mode":493,"modified_unix_seconds":0}}
{"kind":"create_directory","payload":{"id":3,"error":null}}
{"kind":"read_file","payload":{"id":4,"path":"/tmp/results/out.txt"}}
{"kind":"read_file","payload":{"id":4,"contents":"b2sK","error":null}}
{"kind":"execute","payload":{"id":5,"program":"/bin/sh","arguments":["-lc","printf ok"],"current_directory":"/app","environment":[["PATH","/usr/bin:/bin"]],"timeout_millis":60000,"max_output_bytes":8388608}}
{"kind":"execute","payload":{"id":5,"exit_code":0,"stdout":"b2s=","stderr":"","error":null,"timed_out":false,"output_limit_exceeded":false}}
{"kind":"cancel","payload":{"id":6,"target_id":5}}
{"kind":"cancel","payload":{"id":6,"error":null}}
{"kind":"memory","payload":{"id":7}}
{"kind":"memory","payload":{"id":7,"total_kib":786432,"minimum_available_kib":524288,"oom_kills":0,"error":null}}
{"kind":"terminate_tool_processes","payload":{"id":8}}
{"kind":"terminate_tool_processes","payload":{"id":8,"error":null}}
{"kind":"shutdown","payload":{"id":9}}
{"kind":"shutdown","payload":{"id":9,"error":null}}

write_file creates parents and publishes through a sibling temporary file plus rename. read_file accepts only regular files and caps contents at 32 MiB. execute clears the inherited environment, uses only the supplied pairs, captures combined output up to the requested bound, and kills the process group on timeout, output overflow, cancellation, or shutdown. Optional mirror paths receive the same stdout and stderr incrementally but do not alter that bound. execute is a bounded one-response operation rather than a streaming terminal; retained interactive shells use the exec_command/write_stdin tool protocol. memory reports the guest's minimum observed MemAvailable and OOM-kill counter over the session.

Dropping a host request removes its pending response and queues a cancel with a fresh ID. The cancellation queue is bounded by the same 63 admission permits, and a permit is retained until the original request and its cancellation have both been written in that order. Cancelling an unknown or already completed target is successful. The host does not wait for this automatically generated cancel acknowledgement. shutdown stops acceptance, cancels active tool work and shell process groups, gives /bin/sync a five-second deadline, replies, and exits.

Protocol failure

The session fails closed on malformed JSON, an unknown request/response kind, an unknown field in a strict request payload, a frame larger than 64 MiB, a partial frame at EOF, or reuse of an ID that is still active. Clean host EOF cancels active work and exits the guest. A tool response that is too large is replaced with a scoped tool RPC error when that fallback fits; an oversized non-tool response terminates the session. A failed partial response is never turned into a successful tool result.

Egress and lifecycle

[host::EgressLease] is the provider-neutral output of application policy. It combines network mode, guest environment, read-only mounts, public guest files, and host-side guards that must live as long as the VM. The VM crate never resolves secrets or chooses a payment provider. Conflicting environment or mount claims fail closed.

The built-in internet and disabled leases have stable build-cache scopes. Adding provider environment, mounts, or files clears that scope. An application using the resulting lease for Dockerfile builds must assign a non-secret identity with [host::EgressLease::set_build_cache_scope] after composition; otherwise image preparation fails rather than reusing output built through a different route or credential policy.

The default [host::Gvproxy] topology exposes host loopback to the guest at [host::Gvproxy::HOST_IPV4]. If an owned gvproxy exits before cleanup, its status is appended to the caller-selected gvproxy log and emitted through tracing; ordinary owner drop still terminates and reaps a live child.

The last workspace/tool capability kills the VMM child. Workspace startup has a 30-second default deadline covering readiness and egress provisioning; graceful shutdown has a 10-second default covering guest acknowledgement and VMM exit. Both are configurable on [VmWorkspaceBuilder]. Explicit shutdown atomically rejects live sibling capabilities and owner-borrowed requests. Cancelling the shutdown future force-terminates and reaps the child instead of leaving an unreachable VM. Timeouts and request cancellation terminate process groups and descendants.

Cargo features

The default host feature contains image preparation, libkrun lifecycle, and VM-backed tool clients on Linux and macOS. guest-runtime contains only the companion server and the canonical nanocodex-tools workspace runtime. The split exists to produce a small static Linux guest ELF; it is not a second public execution model. Normal native nanocodex-tools and nanocodex-oai-api builds retain their complete default behavior. CI checks the guest-only all-target matrix and builds the actual x86_64 musl guest artifact independently from the host feature.

See docs/VM.md in the repository for CLI operation, egress composition, and build commands.

FilesExpand file tree

README.md

Latest commit

History