fdchannel: under FD exhaustion RecvFD can return the previous receive's FD number and the donated file is silently closed (MSG_CTRUNC never checked)

[alencheung]

BODY

Reading the lisafs FD-donation path at commit cf2f5fb68d1d4 turned up a correctness defect in pkg/fdchannel: if the receiving process is at its FD limit at the moment a donated file descriptor is received, the Linux kernel installs zero FDs, marks the control data truncated, consumes the packet, and drops the donated file's last reference — but reports success. recvFD then validates the receive against the values still sitting in its own reused control buffer from the previous successful receive, and returns the PREVIOUS receive's FD number as if it were the fresh donation. The lisafs channel completes the RPC "successfully" with a wrong FD while the actually-donated file has just been closed by the kernel. Filing as a correctness/robustness bug; no security impact is claimed — every FD that crosses an fd channel belongs to the sandbox's own gofer connection, so the effect stays inside that connection.

The receive path validates against its own reused buffer

• pkg/fdchannel/fdchannel_unsafe.go:43-47: an Endpoint holds ONE msghdr and ONE cmsg buffer for its lifetime; Init (:56-59) allocates the CmsgSpace(sizeofInt32) slice once and never re-zeroes it. The comment at :60-61 documents the assumption: "ep.msghdr.Controllen and ep.cmsg.* are mutated by recvmsg(2), so they're set before calling sendmsg/recvmsg."
• recvFD (:119-138) resets only Controllen (:120-121, to CmsgLen(sizeofInt32) — exactly one FD's worth), issues the syscall (:124/:126), and then checks only three things: the errno (:128), Controllen equality (:131), and cmsg level/type (:134). On success it returns *ep.cmsgData() (:137).
• msghdr.Flags is never read anywhere in the file — the one field the kernel uses to report "control data was dropped" (MSG_CTRUNC) is invisible to this code.

So the validation is only sound if the kernel overwrites the control state on every successful receive. It does not always do so.

What the kernel does when the FD cannot be installed

Verified against torvalds/linux master source as of 2026-09-08 (function names are the stable anchors; line numbers as of that fetch):

1. A SOCK_SEQPACKET receive routes unix_seqpacket_recvmsg -> unix_dgram_recvmsg -> __unix_dgram_recvmsg (net/unix/af_unix.c:2548, :2569). The syscall return value is computed as a byte count BEFORE SCM processing (af_unix.c:2667) — for fdchannel's zero-length packet that is 0, i.e. success.
2. SCM processing is scm_recv_unix (net/core/scm.c:563), which returns void — no failure inside FD installation can reach the syscall result.
3. scm_detach_fds (scm.c:378) computes fdmax = min(scm_max_fds(msg), scm->fp->count) (:383) — 1 for fdchannel's one-FD buffer — and loops scm_recv_one_fd (scm.c:354) over the packet's FDs (:396-400). scm_recv_one_fd allocates the receiver's FD via FD_PREPARE/get_unused_fd_flags (include/linux/file.h:195-204, :217), which fails -EMFILE when the process is at RLIMIT_NOFILE. The loop breaks with i == 0.
4. Every write of cmsg_level/cmsg_type/cmsg_len AND the msg_controllen decrement live inside if (i > 0) (scm.c:402-417). With i == 0: no cmsg is written, msg_controllen is left at the value userspace set, and the control buffer contents are untouched.
5. scm.c:419-420: msg->msg_flags |= MSG_CTRUNC. scm.c:426: __scm_destroy(scm) drops the un-installed file references — the donated file is fput.
6. Back in __unix_dgram_recvmsg: out_free: skb_free_datagram(sk, skb) (af_unix.c:2671-2672) — the packet is consumed. recvmsg returns success.

End-to-end on a live lisafs channel

1. Server side, per RPC (pkg/lisafs/connection.go:171-185): sendFDs(respFDs) queues one zero-length SEQPACKET per FD (pkg/lisafs/channel.go:141-159, one FD per packet via SendFD, fdchannel_unsafe.go:91-103); then closeFDs(respFDs) closes the server's own copies (:178); then marshalHdr(respM, numFDs) with the ACTUAL sent count (:180) and the flipcall wake (:181). By the time the client is woken, the in-flight packet reference is the donated file's last reference.
2. Client side (pkg/lisafs/channel.go:179-200): wake, read numFDs from the header, loop RecvFDNonblock exactly that many times (:190-197).
3. If the sentry process sits at its FD limit at step 2's recvmsg: the kernel installs nothing, sets MSG_CTRUNC, consumes the packet, and closes the donated file — returning success (kernel steps 1-6 above).
4. recvFD's three checks all pass: errno is 0; Controllen still equals CmsgLen(4) because fdchannel itself set it and the kernel never touched it; level/type read back the previous successful receive's SOL_SOCKET/SCM_RIGHTS from the reused buffer.
5. RecvFDNonblock returns the previous receive's FD number. rcvMsg accumulates it via TrackFD (channel.go:196) and the RPC completes with no error (:199). The channel is never marked dead (ch.dead is set only on flipcall errors, channel.go:75) and returns to the pool (pkg/lisafs/client.go:444-453 re-pools anything not dead).

Result: the RPC's caller binds a duplicate of an older FD (e.g. an Open/Walk exchange hands back the wrong open file), while the file the gofer opened for THIS RPC has been closed under it — later operations on the expected file see EBADF/EIO, or silently operate on the wrong file. Nothing is logged in this path: the warning at channel.go:193 fires only when RecvFDNonblock returns an error, which this variant never does.

Note the send-side mirror for contrast: sendFDs documents a deliberate truncation contract (channel.go:136-140 — the i-th failure stops the batch and the header reports the actual count, "the order in which FDs are donated is important"). The receive side has no equivalent: a dropped FD is not detected at all, and even the DETECTED case (a receive error, e.g. on a fresh never-primed Endpoint whose zero-value cmsg fails the level check) is handled by log + break + still returning success (channel.go:192-195, :199) — the RPC completes with fewer FDs than the header promised and the channel stays in the pool. The stale-buffer variant below the warning is strictly worse: wrong FD instead of missing FD.

When this fires (honest preconditions)

• The receiving process — the sentry — must be at its RLIMIT_NOFILE exactly when a channel recvmsg runs. Every gofer-backed open file holds a host FD in the sentry, so an application holding a very large number of files open simultaneously pushes the sentry toward its (raised, but finite) host limit. The window is one syscall wide per donated FD.
• The Endpoint must have completed at least one earlier successful receive, so the reused buffer holds valid-looking values. True after the first FD-carrying RPC on that channel.
• No misbehaving peer is involved — both ends of the channel are gVisor's own code operating exactly as designed.

Reproduction

Found by source reading; the kernel contract above was verified directly against current kernel source, and all gVisor anchors against commit cf2f5fb68d1d4. Runtime reproduction was not executed for this write-up (no Linux container host at hand); both unit lanes below are written to be CI-runnable on Linux.

(a) Kernel-contract demo, plain Go, no gvisor imports: create an AF_UNIX SOCK_SEQPACKET pair; reuse ONE Cmsghdr buffer across receives, mirroring Endpoint. Send FD A (some open file), receive — success, buffer now primed with A. Drop RLIMIT_NOFILE to the current open-FD count. Open file B, send it, close the sender's copy, and receive with the same buffer: the call returns success, the flags word carries MSG_CTRUNC, msg_controllen still equals the input value, and the cmsg data still carries A's number — while B's file has been closed (B's number was never allocated in the receiver; /proc//fd shows it absent, and using the returned "FD" touches file A).

(b) pkg/fdchannel unit test through the Endpoint API: same shape — prime with a successful send/receive, exhaust via setrlimit, SendFD(B) from a helper goroutine, then RecvFDNonblock. Today it returns A's number with a nil error; the assertion for the fixed behavior is that it returns an error instead (see fix below).

(c) End-to-end under runsc (least deterministic, sketched): a container on a lisafs-backed mount holds a very large number of gofer-backed files open while issuing FD-carrying RPCs; the observable is an in-sandbox open that "succeeds" but whose descriptor refers to an unrelated older file, or fails EBADF/EIO moments later. The unit lanes pin the contract deterministically and are the right place to gate it.

Suggested fix

1. pkg/fdchannel recvFD: after the syscall, fail the receive when msghdr.Flags & MSG_CTRUNC != 0 — the flags word is filled in by the kernel and is currently never inspected; on this path (control buffer sized for exactly one FD, senders send exactly one FD per packet) MSG_CTRUNC can only mean "a donated FD was dropped". Optionally also zero the cmsg buffer (or reset cmsg.Level/Type/Len) before each recvmsg, so the level/type validation can never pass against the previous receive's values — defense in depth if the buffer is ever enlarged.
2. pkg/lisafs channel.rcvMsg: a short FD read (err != nil at channel.go:192) should mark the channel dead and fail the RPC instead of log + break + success — a live channel with an unbalanced fd queue must not return to the pool.

Tests either way: lane (a) as a standalone kernel-contract test; lane (b) asserting RecvFD under exhaustion returns an error and never the previous FD; a rcvMsg test asserting the channel is dead after a short FD read; the normal path unchanged.

The repo's existing cmsg-related issues (e.g. #7144 dual-stack PKTINFO, #7012 control-message refactor) are netstack/syscall-emulation surface and unrelated to this host-side channel.

Happy to send a PR with the fix and the tests.

[jdymitarai]

I have submitted a fix in #14729 that clears the control buffer and msghdr.Flags before
ecvmsg, checks ep.msghdr.Flags & unix.MSG_CTRUNC, marks the channel dead and cleans up partially received FDs in lisafs.channel.rcvMsg, and adds regression unit tests in pkg/fdchannel.