Summary
A user-space fork-storm/OOM inside a sandbox kills the toolbox daemon. The in-flight process/execute call then surfaces to the SDK as a bare 502 Bad Gateway from the toolbox proxy, and the sandbox is left in a zombie started state for several minutes before being reconciled to stopped — with errorReason: null and no audit trail. From the API consumer's side this is indistinguishable from a transient proxy blip, which sent our incident investigation in the wrong direction for hours.
Environment
- Daytona Cloud (
app.daytona.io), TypeScript SDK 0.184.0
- Sandbox: default
daytonaio/sandbox:0.8.0 snapshot, 1 vCPU / 1GB / 3GB, daemonVersion v0.201.0-b24aa9b
Reproduction
- Create a 1 vCPU / 1GB sandbox.
sandbox.process.executeCommand(...) a command that forks a large worker pool, e.g. a Python multiprocessing.Pool(64) where each worker allocates tens of MB. (Real-world trigger: agents size pools from nproc, which reports all 64 host cores inside the container because the cgroup limits CPU time, not core visibility.)- ~30–60s in, the exec rejects with a plain 502 from the toolbox proxy.
GET /sandbox/{id} keeps reporting state: "started" for ~2–3 more minutes, then flips to stopped (desiredState: "stopped", errorReason: null).- The in-sandbox daemon log (
/tmp/daytona-daemon.log) ends abruptly mid-request — the last line is the incoming POST /process/execute — with none of the usual Received signal, shutting down gracefully lines that a normal stop writes.
Expected
Some or all of:
- The daemon survives user-space memory pressure — e.g.
oom_score_adj protection or a dedicated memory cgroup, so a user process OOM kills the user process (exit 137 reported through the exec result), not the control plane.
- If the daemon does die, the API surfaces it: an
errorReason like daemon crashed (possible OOM), instead of state: started → silent stopped with no reason.
- Faster health-check reconciliation of a daemonless sandbox (minutes of zombie
started makes the 502 look like a proxy problem).
- Optionally: mask host cores (cpuset) or expose the real CPU allowance so
nproc/os.cpu_count() inside the sandbox reflect the actual limit — that would remove the most common trigger for this class of crash entirely.
Possibly related
Summary
A user-space fork-storm/OOM inside a sandbox kills the toolbox daemon. The in-flight
process/executecall then surfaces to the SDK as a bare 502 Bad Gateway from the toolbox proxy, and the sandbox is left in a zombiestartedstate for several minutes before being reconciled tostopped— witherrorReason: nulland no audit trail. From the API consumer's side this is indistinguishable from a transient proxy blip, which sent our incident investigation in the wrong direction for hours.Environment
app.daytona.io), TypeScript SDK0.184.0daytonaio/sandbox:0.8.0snapshot, 1 vCPU / 1GB / 3GB,daemonVersion v0.201.0-b24aa9bReproduction
sandbox.process.executeCommand(...)a command that forks a large worker pool, e.g. a Pythonmultiprocessing.Pool(64)where each worker allocates tens of MB. (Real-world trigger: agents size pools fromnproc, which reports all 64 host cores inside the container because the cgroup limits CPU time, not core visibility.)GET /sandbox/{id}keeps reportingstate: "started"for ~2–3 more minutes, then flips tostopped(desiredState: "stopped",errorReason: null)./tmp/daytona-daemon.log) ends abruptly mid-request — the last line is the incomingPOST /process/execute— with none of the usualReceived signal, shutting down gracefullylines that a normal stop writes.Expected
Some or all of:
oom_score_adjprotection or a dedicated memory cgroup, so a user process OOM kills the user process (exit 137 reported through the exec result), not the control plane.errorReasonlikedaemon crashed (possible OOM), instead ofstate: started→ silentstoppedwith no reason.startedmakes the 502 look like a proxy problem).nproc/os.cpu_count()inside the sandbox reflect the actual limit — that would remove the most common trigger for this class of crash entirely.Possibly related
started" symptom.