Skip to content

Sandbox CR pins the resources rendered at create; a config change never reaches an existing session #1545

Description

@0xAlcibiades

What happens

A change to the configured sandbox resources does not reach any session that
already exists — not even when its pod is destroyed and recreated.

AgentSandboxBackend::create renders SandboxSpec.resources into the Sandbox
CR's spec.podTemplate. The CR is the durable record, and every pod recreation
re-renders from that stored template: resume only sets spec.replicas = 1,
and the agent-sandbox controller recreates a deleted pod from the same stored
template. Only the cold-create path, where the CR is missing or stopped, picks
up current configuration.

Why it is worse than it sounds

The obvious remedy does not work. Recycling the pod re-renders from the CR, so
the replacement comes back on the old limits; only recreating the CR
migrates the session, and that destroys the workspace it is holding.

So the operator's options are to leave a session on stale limits or to discard
its in-flight work. On a fleet where turns routinely run past two hours, both
are bad, and the sessions that outlive a resources change are the long-running
ones with the largest accumulated state — exactly the ones a memory raise is
meant to protect.

Observed

After raising the agent memory limit, five of thirteen sandboxes were still
running on the old value. One of them was OOM-killed at three and a half hours,
mid-edit and with nothing pushed, on a limit that had already been raised for
every new session. A CR created before the change had its pod destroyed and
recreated well after it, and the new pod came up on the old limit — the CR's
stored spec still pinned it.

Proposed direction

Reconcile the stored template against current configuration for fleet-policy
fields, minimally the agent container's resources:

  • plumb the configured resources into the k8s backend so resume can see them
    (it only receives a sandbox id, not a spec);
  • on resume, patch the stored container resources when they have drifted,
    before setting replicas = 1. That runs while the sandbox is paused and no
    pod exists, so nothing is restarted — the reconciled template is what the next
    pod is built from;
  • leave the rest of the template alone. Harness args, env and principal
    annotations are session identity, and re-rendering them from current config
    would rewrite a session's own setup underneath it.

Worth noting the patch has to be positional rather than a merge: a merge patch
on containers replaces the whole array.

Acceptance

  • A sandbox created under limit X, configuration changed to Y, then paused and
    resumed, comes back under Y without the CR being recreated.
  • Running pods are not restarted by the reconcile.
  • New-sandbox behaviour is unchanged.
  • A container whose name does not match the configured agent container is left
    alone rather than patched by index.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions