panic: UnlinkAt panics the sandbox when creating a overlay whiteout fails

[tanyifeng]

Description

unlink(2) of a copied-up overlay file panics the Sentry and kills the sandbox when creating the matching whiteout fails with ENOENT. Multiple panic logs share the same first line:

panic: unrecoverable overlayfs inconsistency: failed to create whiteout after unlinking upper layer file during UnlinkAt: no such file or directory

Crashing goroutine:

panic: unrecoverable overlayfs inconsistency: failed to create whiteout after unlinking upper layer file during UnlinkAt: no such file or directory

goroutine 42242 gp=0x664741b5860 m=20 mp=0x66473980808 [running]:
panic({0x131c6c0?, 0x6647513cab0?})
	GOROOT/src/runtime/panic.go:879 +0x16f fp=0x66474635400 sp=0x66474635350 pc=0x48c94f
gvisor.dev/gvisor/pkg/sentry/fsimpl/overlay.(*filesystem).UnlinkAt(0x66473141200, {0x16afd78, 0x6647a6ccc88}, 0x664745fc488)
	pkg/sentry/fsimpl/overlay/filesystem.go:1708 +0xa88 fp=0x66474635758 sp=0x66474635400 pc=0xaf8848
gvisor.dev/gvisor/pkg/sentry/vfs.(*VirtualFilesystem).UnlinkAt(0x66473303608, {0x16afd78, 0x6647a6ccc88}, 0x66479d15708?, 0x66473e75d00?)
	pkg/sentry/vfs/vfs.go:755 +0x1ad fp=0x664746357f0 sp=0x66474635758 pc=0x8a8a6d
gvisor.dev/gvisor/pkg/sentry/syscalls/linux.unlinkat(0x6647a6ccc88, 0xffffff9c, 0x66474635950?)
	pkg/sentry/syscalls/linux/sys_file.go:1137 +0x1a8 fp=0x66474635960 sp=0x664746357f0 pc=0xd02e88
gvisor.dev/gvisor/pkg/sentry/syscalls/linux.Unlink(0x6647a6ccc88?, 0x1f?, {{0x55c656cafef8}, {0x0}, {0x1}, {0x0}, {0x0}, {0x55c656a2acd4}})
	pkg/sentry/syscalls/linux/sys_file.go:1124 +0x1d fp=0x66474635988 sp=0x66474635960 pc=0xd02c9d
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).executeSyscall(0x6647a6ccc88, 0x57, {{0x55c656cafef8}, {0x0}, {0x1}, {0x0}, {0x0}, {0x55c656a2acd4}})
	pkg/sentry/kernel/task_syscall.go:143 +0x67d

0xffffff9c is AT_FDCWD; syscall 0x57 is unlink. The panic is in overlay.filesystem.UnlinkAt after the upper-layer file has been removed and whiteout creation (mknod of char 0:0) returns ENOENT.

Steps to reproduce

Copy-up a lower file under a nested overlay, then race unlink of that file (through the overlay) against rmdir of its parent on the raw tmpfs upper (not through the overlay). After overlay removes the upper name, the directory is empty; rmdir can delete the parent before overlay creates the whiteout. Whiteout creation then fails with ENOENT and UnlinkAt panics.

1. runsc (observed)

docker run --rm --runtime=runsc --cap-add=SYS_ADMIN --entrypoint bash \
  gvisor.dev/images/default:latest -c '
set -e
mkdir -p /tmp/l /tmp/uw /tmp/m
mount -t tmpfs tmpfs /tmp/l
mount -t tmpfs tmpfs /tmp/uw
mkdir -p /tmp/uw/upper /tmp/uw/work
mount -t overlay overlay -o lowerdir=/tmp/l,upperdir=/tmp/uw/upper,workdir=/tmp/uw/work /tmp/m
for i in $(seq 0 199); do
  mkdir /tmp/l/d$i
  echo x > /tmp/l/d$i/f
  echo y >> /tmp/m/d$i/f
  echo "round $i: unlink /tmp/m/d$i/f"
  # Python unlink + rmdir threads: tight syscalls to hit the whiteout race.
  python3 -c "
import os, threading
stop = threading.Event()
def spin():
    while not stop.is_set():
        try: os.rmdir(\"/tmp/uw/upper/d$i\")
        except OSError: pass
ts = [threading.Thread(target=spin, daemon=True) for _ in range(4)]
[t.start() for t in ts]
try: os.unlink(\"/tmp/m/d$i/f\")
except OSError as e: print(\"unlink failed but alive:\", e)
stop.set()
for t in ts: t.join()
"
done
echo "no panic after 200 rounds (sandbox still alive)"
'

round 27: unlink /tmp/m/d27/f
round 28: unlink /tmp/m/d28/f
round 29: unlink /tmp/m/d29/f

The process never prints sandbox still alive. docker run exits 2. First line of the Sentry panic log is identical to production:

panic: unrecoverable overlayfs inconsistency: failed to create whiteout after unlinking upper layer file during UnlinkAt: no such file or directory

2. runc (Linux overlayfs)

Same command with --runtime=runc.

round 199: unlink /tmp/m/d199/f
no panic after 200 rounds (sandbox still alive)

Exit code 0. The container stays up.

Linux overlayfs does not unlink the upper name and then mknod a whiteout. ovl_cleanup_and_whiteout() creates the whiteout in the overlay workdir, then renames it over the upper file (directories use RENAME_EXCHANGE). The parent still has that name for the whole operation, so a concurrent rmdir of the parent cannot observe an empty directory. If the rename fails, the kernel returns an errno; it does not panic.

gVisor UnlinkAt is unlink-then-mknod. That is not atomic with respect to other users of the upper filesystem (the overlay RenameAt path already notes that essentially no gVisor filesystem supports RENAME_WHITEOUT). After the unlink, the upper dir can be empty; rmdir on the raw tmpfs upper can remove the parent; whiteout mknod then returns ENOENT; UnlinkAt panics.

runsc version

runsc version release-20260824.0-93-gb3674d543f
spec: 1.2.1

docker version (if using docker)

Client: Docker Engine - Community 26.1.3
Server: Docker Engine - Community 26.1.3
 containerd: 1.6.32

uname

Linux 6.6.110

kubectl (if using Kubernetes)

repo state (if built from source)

release-20260824.0-93-gb3674d543f

runsc debug logs (if available)

Crashing goroutine only:

panic: unrecoverable overlayfs inconsistency: failed to create whiteout after unlinking upper layer file during UnlinkAt: no such file or directory

goroutine 42242 gp=0x664741b5860 m=20 mp=0x66473980808 [running]:
panic({0x131c6c0?, 0x6647513cab0?})
	GOROOT/src/runtime/panic.go:879 +0x16f fp=0x66474635400 sp=0x66474635350 pc=0x48c94f
gvisor.dev/gvisor/pkg/sentry/fsimpl/overlay.(*filesystem).UnlinkAt(0x66473141200, {0x16afd78, 0x6647a6ccc88}, 0x664745fc488)
	pkg/sentry/fsimpl/overlay/filesystem.go:1708 +0xa88 fp=0x66474635758 sp=0x66474635400 pc=0xaf8848
gvisor.dev/gvisor/pkg/sentry/vfs.(*VirtualFilesystem).UnlinkAt(0x66473303608, {0x16afd78, 0x6647a6ccc88}, 0x66479d15708?, 0x66473e75d00?)
	pkg/sentry/vfs/vfs.go:755 +0x1ad fp=0x664746357f0 sp=0x66474635758 pc=0x8a8a6d
gvisor.dev/gvisor/pkg/sentry/syscalls/linux.unlinkat(0x6647a6ccc88, 0xffffff9c, 0x66474635950?)
	pkg/sentry/syscalls/linux/sys_file.go:1137 +0x1a8 fp=0x66474635960 sp=0x664746357f0 pc=0xd02e88
gvisor.dev/gvisor/pkg/sentry/syscalls/linux.Unlink(0x6647a6ccc88?, 0x1f?, {{0x55c656cafef8}, {0x0}, {0x1}, {0x0}, {0x0}, {0x55c656a2acd4}})
	pkg/sentry/syscalls/linux/sys_file.go:1124 +0x1d fp=0x66474635988 sp=0x66474635960 pc=0xd02c9d
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).executeSyscall(0x6647a6ccc88, 0x57, {{0x55c656cafef8}, {0x0}, {0x1}, {0x0}, {0x0}, {0x55c656a2acd4}})
	pkg/sentry/kernel/task_syscall.go:143 +0x67d fp=0x66474635cc8 sp=0x66474635988 pc=0xb6913d
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscallInvoke(0x6647a6ccc88, 0x57, {{0x55c656cafef8}, {0x0}, {0x1}, {0x0}, {0x0}, {0x55c656a2acd4}})
	pkg/sentry/kernel/task_syscall.go:323 +0x4b fp=0x66474635d20 sp=0x66474635cc8 pc=0xb6a2cb
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscallEnter(0x6647a6ccc88, 0x57, {{0x55c656cafef8}, {0x0}, {0x1}, {0x0}, {0x0}, {0x55c656a2acd4}})
	pkg/sentry/kernel/task_syscall.go:283 +0x68 fp=0x66474635d70 sp=0x66474635d20 pc=0xb69fe8
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscall(0x66474316dc0?)
	pkg/sentry/kernel/task_syscall.go:258 +0x2a5 fp=0x66474635e48 sp=0x66474635d70 pc=0xb69d65
gvisor.dev/gvisor/pkg/sentry/kernel.(*runApp).execute(0x664743895c0?, 0x6647a6ccc88)
	pkg/sentry/kernel/task_run.go:258 +0xf9c fp=0x66474635f30 sp=0x66474635e48 pc=0xb5e7fc
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).run(0x6647a6ccc88, 0x2b6d)
	pkg/sentry/kernel/task_run.go:89 +0x202 fp=0x66474635fc0 sp=0x66474635f30 pc=0xb5d362
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).Start.gowrap1()
	pkg/sentry/kernel/task_start.go:566 +0x1b fp=0x66474635fe0 sp=0x66474635fc0 pc=0xb67c5b
runtime.goexit({})
	src/runtime/asm_amd64.s:1771 +0x1 fp=0x66474635fe8 sp=0x66474635fe0 pc=0x495601
created by gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).Start in goroutine 42119
	pkg/sentry/kernel/task_start.go:566 +0xc5

[ayushr2]

@tanyifeng I think #14073 is fixing this?

[jdymitarai]

I investigated this issue and verified that PR #14073 does not address this: PR #14073 strictly covers directory \RenameAt\ and opaque xattrs in user namespaces (fixing #14033), and does not touch \UnlinkAt, \RmdirAt, or whiteout creation failures.

I have submitted PR #14727 which replaces the panics during whiteout creation/recreation with warning logs and returns the error gracefully to userspace without panicking the sandbox, along with a regression test in \mount.cc.

[tanyifeng]

As jdymitarai said, this is a new issue, and I haven't thought of a good solution yet.

@tanyifeng I think #14073 is fixing this?