Configure adequately sized /dev/shm for singleuser pods to avoid silent failures in Ray, PyTorch DataLoader, and other shm-using libraries #54

Problem

Kubernetes containers default to a 64 MiB /dev/shm tmpfs (the historical Docker/containerd default — 67108864 bytes), which is far below what common data-science libraries need. Several workloads that users routinely run inside JupyterLab fail or degrade silently:

• Ray — ray.init() emits the warning below and silently falls back to disk-backed /tmp for the plasma object store, eliminating zero-copy shared memory and significantly slowing any code that passes objects between actors/tasks:

  WARNING services.py:2168 -- WARNING: The object store is using /tmp instead of /dev/shm
  because /dev/shm has only 67108864 bytes available. This will harm performance!
  ... Make sure to set this to more than 30% of available RAM.
  

• PyTorch DataLoader(num_workers>0) — worker processes share batches with the main process via shared-memory tensors. With 64 MiB they hit RuntimeError: DataLoader worker (pid X) is killed by signal: Bus error once batches exceed the limit. This is a well-known pitfall and a very bad first-experience for anyone running standard PyTorch tutorials in the notebook.
• Python multiprocessing.shared_memory, mp.Array, mp.Queue — same constraint.

This affects every singleuser pod the chart spawns, and each affected user has to either rediscover the workaround or contact their operator.

Upstream documentation context

Ray's docs acknowledge the issue but only in the VM-cluster best-practices guide:

"By default, Ray will try to use /dev/shm for the object store, but if it is not large enough … Ray will write the plasma store to disk instead, which may cause significant performance problems."

The fix it suggests (--shm-size to docker run) does not apply on Kubernetes — there is no shmSize field on a container. The standard workaround is mounting a memory-backed emptyDir at /dev/shm.

Why a naive singleuser.extraVolumes value doesn't fix this in the current chart

config/jupyterhub/01-spawner.py sets c.KubeSpawner.volumes and c.KubeSpawner.volume_mounts directly to lists (for the home PVC and the nebi-bin volume). Anything provided through z2jh's singleuser.extraVolumes/extraVolumeMounts is overridden by these direct assignments. So operators cannot fix this with values alone — the chart has to either append to those lists in spawner config or expose a dedicated values surface for shared memory.

Proposal

Add chart-level support for a memory-backed /dev/shm mount on singleuser pods, configurable both globally and per profile.

1. Chart-wide default (e.g. in values.yaml):

singleuser:
  sharedMemory:
    enabled: true
    sizeLimit: 8Gi   # >= 30% of pod memory limit; counts against pod memory cgroup

2. Per-profile override via kubespawner_override, since the chart's profileList already differentiates CPU vs GPU profiles that have very different memory limits:

profileList:
  - display_name: "GPU (2x NVIDIA P100)"
    kubespawner_override:
      shm_size_limit: 16Gi   # rendered into the dshm emptyDir for this profile only

3. Implementation — extend 01-spawner.py to append to c.KubeSpawner.volumes/volume_mounts when singleuser.sharedMemory.enabled is true, and read the per-profile override if present. Equivalent to:

volumes:
  - name: dshm
    emptyDir:
      medium: Memory
      sizeLimit: 8Gi
volumeMounts:
  - name: dshm
    mountPath: /dev/shm

4. Documentation — note in the values file that:

• medium: Memory emptyDir counts against the pod's memory cgroup, so a 32 GiB profile with 8 GiB shm has ~24 GiB left for everything else.
• Recommended sizing: sizeLimit ≥ 30% of the pod's memory limit (Ray's plasma default).

Verification

After deploy, spawn a fresh user pod and run:

kubectl -n jupyterhub exec <jupyter-pod> -- df -h /dev/shm
# Filesystem      Size  Used Avail Use% Mounted on
# tmpfs           8.0G    0  8.0G   0% /dev/shm

In a notebook:

import ray; ray.init()  # no /dev/shm warning

PyTorch DataLoader(num_workers=4) over a non-trivial dataset should run without Bus error.

Workaround (current)

Operators can patch this from values today via hub.extraConfig, which loads after 01-spawner.py:

hub:
  extraConfig:
    99-shm: |
      c.KubeSpawner.volumes = list(c.KubeSpawner.volumes) + [
          {"name": "dshm", "emptyDir": {"medium": "Memory", "sizeLimit": "8Gi"}},
      ]
      c.KubeSpawner.volume_mounts = list(c.KubeSpawner.volume_mounts) + [
          {"name": "dshm", "mountPath": "/dev/shm"},
      ]

This unblocks operators but is the kind of workaround every deployment will have to rediscover. Folding it into the chart with a clean values surface avoids that.

Related

• Companion issue against nebari-rayserve-pack: Configure adequately sized /dev/shm for Ray pods to avoid plasma store falling back to disk rayserve-pack#9

References

• Best practices for deploying large clusters — Ray docs (closest upstream guidance — VM-cluster only)
• Debugging Memory Issues — Ray docs
• PyTorch DataLoader bus-error issue (#2244) — canonical reference for the num_workers>0 failure mode