cuda_ipc: VMM/mempool memory with POSIX_FD-only handle types silently loses the cuda_ipc lane (PyTorch expandable_segments) — add a pidfd-based POSIX_FD arm

[xyuzh]

Describe the feature

cuda_ipc currently exports VMM (cuMemCreate) and mempool (cudaMallocAsync) memory only via CU_MEM_HANDLE_TYPE_FABRIC. Allocations whose allowed_handle_types is CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR — which is what PyTorch produces by default — are classified UCT_CUDA_IPC_KEY_HANDLE_TYPE_NO_IPC and silently lose the cuda_ipc lane. Request: add a POSIX_FD arm to the existing handle-type dispatch, using pidfd_getfd(2) on the importer side so the file descriptor can be retrieved without a side channel.

The dispatch in src/uct/cuda/cuda_ipc/cuda_ipc_md.c today:

if (legacy_capable)
    goto legacy_path;                                     /* cudaMalloc -> cuIpcGetMemHandle */
if (!(allowed_handle_types & CU_MEM_HANDLE_TYPE_FABRIC))
    goto non_ipc;                                         /* <- POSIX_FD-only allocations land here */
...cuMemExportToShareableHandle(..., CU_MEM_HANDLE_TYPE_FABRIC)...
non_ipc:
    key->ph.handle_type = UCT_CUDA_IPC_KEY_HANDLE_TYPE_NO_IPC;

Since allowed_handle_types is fixed at cuMemCreate time by the allocator, UCX cannot upgrade such memory to fabric after the fact — and fabric-typed allocations require fabricmanager/IMEX, which most single-node and cloud deployments do not have.

Motivation / real-world impact

PyTorch with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (a very common fragmentation mitigation in large-model training) allocates via VMM with requestedHandleTypes = CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR (it auto-selects FABRIC only when the device reports fabric access). Result on a same-node H100/NVLink system, one-sided READ of a 512 MiB buffer between two processes (via NIXL -> UCX, UCX_PROTO_INFO=y):

Source allocation	Steady-state READ	Selected proto
default cudaMalloc	179-201 GB/s	zero-copy | cuda_ipc/cuda
expandable_segments:True	0.29-0.32 GB/s	software emulation | tcp

A single allocator env var causes a silent ~600x degradation (data remains correct), which is brutal to diagnose: registration succeeds and nothing is logged above trace level. We bisected this while debugging RL weight-sync transfers through Ray's RDT/NIXL integration; details in ai-dynamo/nixl#1754 (a one-time warning is proposed in ai-dynamo/nixl#1755, but a warning only diagnoses — this feature would fix it). The same gap applies to cudaMallocAsync mempool memory on non-fabric machines, since the mempool arm is also FABRIC-only.

Proposed design

PyTorch already ships exactly this protocol for its own IPC path (c10/cuda/CUDACachingAllocator.cpp, ExpandableSegment::share / fromShared), which can serve as prior art:

1. Pack (exporter): when !legacy_capable and allowed_handle_types & CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR, call cuMemRetainAllocationHandle + cuMemExportToShareableHandle(&fd, handle, CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR, 0) and pack {pid, fd} (plain bytes) in the rkey under a new UCT_CUDA_IPC_KEY_HANDLE_TYPE_POSIX_FD. Keep the FD open until deregistration.
2. Unpack (importer, same node — already guaranteed by cuda_ipc reachability): pidfd_open(pid) + pidfd_getfd(pidfd, fd) to obtain a local duplicate, then cuMemImportFromShareableHandle + cuMemMap/cuMemSetAccess, cached like the existing VMM/mempool mappings.
3. Fallbacks: pidfd_getfd requires PTRACE_MODE_ATTACH_REALCREDS (same-user processes, the typical colocated ML case) and Linux >= 5.6; on EPERM/ENOSYS, fall back to the current NO_IPC behavior — ideally with a one-time diag/warn-level message instead of trace, so the degradation is visible.

The rkey union/enum/dispatch structure already accommodates multiple handle types, so this slots into the existing design rather than changing it.

Environment where this was hit

• UCX bundled with nixl 1.2.0 (HAVE_CUDA_FABRIC enabled), CUDA 12.9, driver-side fabric access unavailable (no fabricmanager/IMEX)
• PyTorch 2.11, Ray 2.55.1 RDT (ray.put(_tensor_transport="nixl"))
• 8x H100 (NVLink), single node, AWS EKS

Happy to share the self-contained two-process reproducer scripts and full bisect data (CUDA_VISIBLE_DEVICES masking, UCX_TLS/RNDV overrides, NCCL presence etc. were all ruled out; the allocator setting alone reproduces it).

[xyuzh]

#11183