What happened?
Workspace pods are created with no ephemeral-storage request unless every individual Stack sets one in spec.workspaceTemplate.spec.podTemplate. The share emptyDir holds node_modules and the downloaded provider plugins, which for a small TypeScript program is already 1.3–2.4 GiB. The scheduler can't see any of that, so it packs workspace pods onto a node until nodefs runs out.
We hit both resulting failure modes on a cluster running PKO 2.7.0, dozens of Stacks, one Stack per tenant app:
-
Eviction. In 24h, 12 nodes flapped DiskPressure and 33 distinct workspace pods were evicted. kubelet picks the pod most exceeding its request, and a pod with a zero request and 2 GiB of usage always wins — frequently one with an in-flight pulumi up. The interrupted update leaves pending operations that need an interactive pulumi refresh, so one eviction wedges that stack indefinitely.
-
Plain ENOSPC mid-install. 14 workspace pods landed on one 52.94 GiB node within 6 minutes; node volume free space went 23 GiB → 0 in under 5 minutes:
error: installing plugin; run `pulumi plugin install resource aws v7.42.0` to retry manually:
untarring file /home/pulumi/.pulumi/plugins/resource-aws-v7.42.0/pulumi-resource-aws:
write ...: no space left on device
This one is worse than a normal eviction, because it doesn't self-heal. npm install reported success but wrote an incomplete tree, so every later up in that same long-lived workspace pod fails with:
Error: Cannot find module './util'
Require stack:
- /share/source/<dir>/node_modules/source-map/lib/source-map-generator.js
- ... /node_modules/@pulumi/pulumi/cmd/run/index.js
The workspace pod isn't recreated on retry and Install only re-runs when the source revision changes, so retries call Up against the same broken node_modules forever — 8 stacks sat on identical failures for 3 days across 10 retry attempts each. Recovery is kubectl delete pod <stack>-workspace-0.
Example
There's no example program for this, as it can happen on any program that runs on a crowded node with no disk space requests.
Output of pulumi about
N/A
Additional context
Expected behavior
An operator- or chart-level default for the workspace pod template, so ephemeral-storage requests can be set once instead of per Stack. Concretely, either:
- a chart value like workspace.defaultPodTemplate / workspace.resources that the operator merges into every Workspace it creates, or
- a built-in non-zero default ephemeral-storage request for the pulumi container (with an override), given that a share volume containing
node_modules plus provider plugins is never free.
Per-Stack configuration doesn't cover this case: our Stack CRs are generated by a controller, so users of that platform can't edit them, and the same should apply to Kratix Promises, Crossplane compositions or ArgoCD-generated Stacks. Anything not expressible as a template default has to be re-added by whoever generates the Stack.
Workaround
An AdmissionController Policy (eg. Kyverno ClusterPolicy) mutating every pulumi.com/v1/Stack in the operator namespace to inject requests.ephemeral-storage: 4Gi / limits: 8Gi into spec.workspaceTemplate.spec.podTemplate.spec.containers[name=pulumi]. It works, but it needs a cluster-wide admission controller to patch a field the operator could default itself, and it only applies to Stacks as they're created or updated.
Prior work
Found the following issues:
Contributing
Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
What happened?
Workspace pods are created with no ephemeral-storage request unless every individual Stack sets one in
spec.workspaceTemplate.spec.podTemplate. TheshareemptyDir holdsnode_modulesand the downloaded provider plugins, which for a small TypeScript program is already 1.3–2.4 GiB. The scheduler can't see any of that, so it packs workspace pods onto a node until nodefs runs out.We hit both resulting failure modes on a cluster running PKO 2.7.0, dozens of Stacks, one Stack per tenant app:
Eviction. In 24h, 12 nodes flapped
DiskPressureand 33 distinct workspace pods were evicted.kubeletpicks the pod most exceeding its request, and a pod with a zero request and 2 GiB of usage always wins — frequently one with an in-flightpulumi up. The interrupted update leaves pending operations that need an interactivepulumi refresh, so one eviction wedges that stack indefinitely.Plain ENOSPC mid-install. 14 workspace pods landed on one 52.94 GiB node within 6 minutes; node volume free space went 23 GiB → 0 in under 5 minutes:
This one is worse than a normal eviction, because it doesn't self-heal.
npm installreported success but wrote an incomplete tree, so every later up in that same long-lived workspace pod fails with:The workspace pod isn't recreated on retry and Install only re-runs when the source revision changes, so retries call Up against the same broken node_modules forever — 8 stacks sat on identical failures for 3 days across 10 retry attempts each. Recovery is
kubectl delete pod <stack>-workspace-0.Example
There's no example program for this, as it can happen on any program that runs on a crowded node with no disk space requests.
Output of
pulumi aboutN/A
Additional context
Expected behavior
An operator- or chart-level default for the workspace pod template, so ephemeral-storage requests can be set once instead of per Stack. Concretely, either:
node_modulesplus provider plugins is never free.Per-Stack configuration doesn't cover this case: our Stack CRs are generated by a controller, so users of that platform can't edit them, and the same should apply to Kratix Promises, Crossplane compositions or ArgoCD-generated Stacks. Anything not expressible as a template default has to be re-added by whoever generates the Stack.
Workaround
An AdmissionController Policy (eg. Kyverno ClusterPolicy) mutating every pulumi.com/v1/Stack in the operator namespace to inject
requests.ephemeral-storage: 4Gi / limits: 8Gi intospec.workspaceTemplate.spec.podTemplate.spec.containers[name=pulumi]. It works, but it needs a cluster-wide admission controller to patch a field the operator could default itself, and it only applies to Stacks as they're created or updated.Prior work
Found the following issues:
storage"); never implemented.
go builduse of disk space leads to pod eviction #381 — same class of failure against the v1 operator pod.Contributing
Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).