Skip to content

runner: surface sandbox file-descriptor exhaustion as a degraded state #4996

Description

@mu-hashmi

Is your feature request related to a problem? Please describe.

When the host-side filesystem path backing a sandbox runs out of file handles, in-sandbox behavior degrades in confusing ways: new processes fail at the dynamic loader (error while loading shared libraries), toolbox exec and PTY calls fail, and file operations return EMFILE-family errors. Throughout, the sandbox stays started with no error state and no errorReason. Users debug their own code while the platform is the cause. Same principle as #4991: failures must leave evidence as explicit state, not vanish into generic errors.

Describe the solution you'd like

Classify fd-exhaustion signatures (EMFILE/ENFILE, "too many open files", loader failures on exec paths) from the runner's and daemon's view of the sandbox, and surface a degraded condition on the sandbox with an errorReason while it persists, clearing automatically on recovery.

One hard constraint: this classification must be surfacing-only. The existing recoverable-error patterns drive automated recovery actions (the storage-limit flow recreates the container); fd exhaustion must not be wired into that path. Restarting or recreating a sandbox against an exhausted handle table amplifies the outage instead of ending it, and these episodes typically self-resolve once writeback drains.

Acceptance criteria:

  • fd-exhaustion signatures are classified distinctly from storage-limit recoverable errors
  • Sandbox exposes a degraded condition with errorReason while exhaustion persists, cleared on recovery
  • No automated restart/recreate is triggered by this classification
  • SDK/dashboard can read the condition (no silent started-while-broken)

Describe alternatives you've considered

Auto-recovery via restart (rejected: restart storms against a full handle table). Status quo (users misattribute platform degradation to their own workloads).

Additional context

Related: #4991 establishes the same surfacing pattern for snapshot capture failures.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions