What happens
A change to the configured sandbox resources does not reach any session that
already exists — not even when its pod is destroyed and recreated.
AgentSandboxBackend::create renders SandboxSpec.resources into the Sandbox
CR's spec.podTemplate. The CR is the durable record, and every pod recreation
re-renders from that stored template: resume only sets spec.replicas = 1,
and the agent-sandbox controller recreates a deleted pod from the same stored
template. Only the cold-create path, where the CR is missing or stopped, picks
up current configuration.
Why it is worse than it sounds
The obvious remedy does not work. Recycling the pod re-renders from the CR, so
the replacement comes back on the old limits; only recreating the CR
migrates the session, and that destroys the workspace it is holding.
So the operator's options are to leave a session on stale limits or to discard
its in-flight work. On a fleet where turns routinely run past two hours, both
are bad, and the sessions that outlive a resources change are the long-running
ones with the largest accumulated state — exactly the ones a memory raise is
meant to protect.
Observed
After raising the agent memory limit, five of thirteen sandboxes were still
running on the old value. One of them was OOM-killed at three and a half hours,
mid-edit and with nothing pushed, on a limit that had already been raised for
every new session. A CR created before the change had its pod destroyed and
recreated well after it, and the new pod came up on the old limit — the CR's
stored spec still pinned it.
Proposed direction
Reconcile the stored template against current configuration for fleet-policy
fields, minimally the agent container's resources:
- plumb the configured resources into the k8s backend so
resume can see them
(it only receives a sandbox id, not a spec);
- on
resume, patch the stored container resources when they have drifted,
before setting replicas = 1. That runs while the sandbox is paused and no
pod exists, so nothing is restarted — the reconciled template is what the next
pod is built from;
- leave the rest of the template alone. Harness args, env and principal
annotations are session identity, and re-rendering them from current config
would rewrite a session's own setup underneath it.
Worth noting the patch has to be positional rather than a merge: a merge patch
on containers replaces the whole array.
Acceptance
- A sandbox created under limit X, configuration changed to Y, then paused and
resumed, comes back under Y without the CR being recreated.
- Running pods are not restarted by the reconcile.
- New-sandbox behaviour is unchanged.
- A container whose name does not match the configured agent container is left
alone rather than patched by index.
What happens
A change to the configured sandbox resources does not reach any session that
already exists — not even when its pod is destroyed and recreated.
AgentSandboxBackend::createrendersSandboxSpec.resourcesinto the SandboxCR's
spec.podTemplate. The CR is the durable record, and every pod recreationre-renders from that stored template:
resumeonly setsspec.replicas = 1,and the agent-sandbox controller recreates a deleted pod from the same stored
template. Only the cold-create path, where the CR is missing or stopped, picks
up current configuration.
Why it is worse than it sounds
The obvious remedy does not work. Recycling the pod re-renders from the CR, so
the replacement comes back on the old limits; only recreating the CR
migrates the session, and that destroys the workspace it is holding.
So the operator's options are to leave a session on stale limits or to discard
its in-flight work. On a fleet where turns routinely run past two hours, both
are bad, and the sessions that outlive a resources change are the long-running
ones with the largest accumulated state — exactly the ones a memory raise is
meant to protect.
Observed
After raising the agent memory limit, five of thirteen sandboxes were still
running on the old value. One of them was OOM-killed at three and a half hours,
mid-edit and with nothing pushed, on a limit that had already been raised for
every new session. A CR created before the change had its pod destroyed and
recreated well after it, and the new pod came up on the old limit — the CR's
stored spec still pinned it.
Proposed direction
Reconcile the stored template against current configuration for fleet-policy
fields, minimally the agent container's
resources:resumecan see them(it only receives a sandbox id, not a spec);
resume, patch the stored container resources when they have drifted,before setting
replicas = 1. That runs while the sandbox is paused and nopod exists, so nothing is restarted — the reconciled template is what the next
pod is built from;
annotations are session identity, and re-rendering them from current config
would rewrite a session's own setup underneath it.
Worth noting the patch has to be positional rather than a merge: a merge patch
on
containersreplaces the whole array.Acceptance
resumed, comes back under Y without the CR being recreated.
alone rather than patched by index.