`cuda_copy` transport in `UCX_TLS` degrades LLM inference throughput

[UnpureRationalist]

Summary

Adding cuda_copy to the UCX_TLS allowlist causes a significant drop in LLM inference throughput (requests-per-second) in our serving runtime. When cuda_copy is excluded from UCX_TLS, QPS returns to baseline. We'd like the maintainers' input on where in the cuda_copy transport this overhead might originate and how to keep cuda_copy available as a fallback without the throughput penalty.

Environment

• UCX version & Transport info:

  $ ucx_info -v
  # Library version: 1.19.1
  # Library path: /usr/local/lib/libucs.so.0
  # API headers version: 1.19.1
  # Git branch 'v1.19.x', revision 812388c
  # Configured with: --disable-logging --disable-debug --disable-assertions --disable-params-check --enable-mt --prefix=/usr/local --enable-shared --disable-static --disable-doxygen-doc --enable-optimizations --enable-cma --enable-devel-headers --with-cuda=/usr/local/cuda --with-verbs --with-dm
  
  $ ucx_info -d | grep Transport
  #      Transport: self
  #      Transport: tcp
  #      Transport: tcp
  #      Transport: tcp
  #      Transport: tcp
  #      Transport: tcp
  #      Transport: tcp
  #      Transport: tcp
  #      Transport: tcp
  #      Transport: tcp
  #      Transport: tcp
  #      Transport: tcp
  #      Transport: tcp
  #      Transport: sysv
  #      Transport: posix
  #      Transport: cuda_copy
  #      Transport: cuda_ipc
  #      Transport: rc_verbs
  #      Transport: ud_verbs
  #      Transport: dc_mlx5
  #      Transport: rc_mlx5
  #      Transport: ud_mlx5
  #      Transport: rc_verbs
  #      Transport: ud_verbs
  #      Transport: dc_mlx5
  #      Transport: rc_mlx5
  #      Transport: ud_mlx5
  #      Transport: rc_verbs
  #      Transport: ud_verbs
  #      Transport: dc_mlx5
  #      Transport: rc_mlx5
  #      Transport: ud_mlx5
  #      Transport: rc_verbs
  #      Transport: ud_verbs
  #      Transport: dc_mlx5
  #      Transport: rc_mlx5
  #      Transport: ud_mlx5
  #      Transport: rc_verbs
  #      Transport: ud_verbs
  #      Transport: dc_mlx5
  #      Transport: rc_mlx5
  #      Transport: ud_mlx5
  #      Transport: rc_verbs
  #      Transport: ud_verbs
  #      Transport: dc_mlx5
  #      Transport: rc_mlx5
  #      Transport: ud_mlx5
  #      Transport: rc_verbs
  #      Transport: ud_verbs
  #      Transport: dc_mlx5
  #      Transport: rc_mlx5
  #      Transport: ud_mlx5
  #      Transport: rc_verbs
  #      Transport: ud_verbs
  #      Transport: dc_mlx5
  #      Transport: rc_mlx5
  #      Transport: ud_mlx5
  #      Transport: cma

• Host CUDA driver / runtime: Driver version 580.95.05 & CUDA 13.0
• Docker Container CUDA version: CUDA 12.9
• GPU: H200 * 8
• OS / kernel:

  # Host
  $ cat /etc/os-release
  NAME="Rocky Linux"
  VERSION="9.3 (Blue Onyx)"
  ID="rocky"
  ID_LIKE="rhel centos fedora"
  VERSION_ID="9.3"
  PLATFORM_ID="platform:el9"
  PRETTY_NAME="Rocky Linux 9.3 (Blue Onyx)"
  ANSI_COLOR="0;32"
  LOGO="fedora-logo-icon"
  CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
  HOME_URL="https://rockylinux.org/"
  BUG_REPORT_URL="https://bugs.rockylinux.org/"
  SUPPORT_END="2032-05-31"
  ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
  ROCKY_SUPPORT_PRODUCT_VERSION="9.3"
  REDHAT_SUPPORT_PRODUCT="Rocky Linux"
  REDHAT_SUPPORT_PRODUCT_VERSION="9.3"
  
  # Container
  $ cat /etc/os-release
  PRETTY_NAME="Ubuntu 24.04.2 LTS"
  NAME="Ubuntu"
  VERSION_ID="24.04"
  VERSION="24.04.2 LTS (Noble Numbat)"
  VERSION_CODENAME=noble
  ID=ubuntu
  ID_LIKE=debian
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  UBUNTU_CODENAME=noble
  LOGO=ubuntu-logo

• Interconnect:

  $ nvidia-smi topo -m
          GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    CPU Affinity    NUMA Affinity   GPU NUMA ID
  GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    PIX     PIX     NODE    NODE    SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
  GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    PIX     PIX     NODE    NODE    SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
  GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    NODE    NODE    PIX     PIX     SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
  GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    NODE    NODE    PIX     PIX     SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
  GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     SYS     SYS     PIX     PIX     NODE    NODE    56-111,168-223  1               N/A
  GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     SYS     SYS     PIX     PIX     NODE    NODE    56-111,168-223  1               N/A
  GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     SYS     SYS     NODE    NODE    PIX     PIX     56-111,168-223  1               N/A
  GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     SYS     SYS     NODE    NODE    PIX     PIX     56-111,168-223  1               N/A
  NIC0    PIX     PIX     NODE    NODE    SYS     SYS     SYS     SYS      X      PIX     NODE    NODE    SYS     SYS     SYS     SYS
  NIC1    PIX     PIX     NODE    NODE    SYS     SYS     SYS     SYS     PIX      X      NODE    NODE    SYS     SYS     SYS     SYS
  NIC2    NODE    NODE    PIX     PIX     SYS     SYS     SYS     SYS     NODE    NODE     X      PIX     SYS     SYS     SYS     SYS
  NIC3    NODE    NODE    PIX     PIX     SYS     SYS     SYS     SYS     NODE    NODE    PIX      X      SYS     SYS     SYS     SYS
  NIC4    SYS     SYS     SYS     SYS     PIX     PIX     NODE    NODE    SYS     SYS     SYS     SYS      X      PIX     NODE    NODE
  NIC5    SYS     SYS     SYS     SYS     PIX     PIX     NODE    NODE    SYS     SYS     SYS     SYS     PIX      X      NODE    NODE
  NIC6    SYS     SYS     SYS     SYS     NODE    NODE    PIX     PIX     SYS     SYS     SYS     SYS     NODE    NODE     X      PIX
  NIC7    SYS     SYS     SYS     SYS     NODE    NODE    PIX     PIX     SYS     SYS     SYS     SYS     NODE    NODE    PIX      X 
  
  Legend:
  
  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks
  
  NIC Legend:
  
  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7

• Runtime that consumes UCX: an in-house LLM inference engine that uses NIXL on top of UCX for P2P KV Cache transfer. In our benchmark, only one process runs on one GPU and the code execution path does not touch real RDMA Read/Write: we just init nixlAgent with the ucx backend and register a DRAM region and a VRAM region, then the QPS of inference decreases noticeably. The real KV Cache transfer path goes through the cudaMemcpyAsync and cudaMemcpy2DAsync APIs without any RDMA operation.

Setup

We run a benchmark that sends a fixed batch of inference requests to the runtime and measures end-to-end throughput. The only variable between the two configurations below is whether cuda_copy is allowed in UCX_TLS.

• Baseline (no cuda_copy):

  export UCX_TLS=ib,tcp,self,sm
  # benchmark invocation

• With cuda_copy:

  export UCX_TLS=ib,tcp,self,sm,cuda_copy  # or simply default (no restriction)
  # same benchmark invocation

All other UCX_* environment variables are identical between the two runs.

Observations

Run	UCX_TLS	QPS (req/s)
Baseline	ib,tcp,self,sm (cuda_copy excluded)	2952
cuda_copy enabled	ib,tcp,self,sm,cuda_copy (cuda_copy allowed)	2411

In our runs the QPS drop is roughly 18%. The regression is fully reproducible and only depends on whether cuda_copy is in the TLS list.

What we confirmed

1. The regression disappears when cuda_copy is excluded from UCX_TLS. Toggling UCX_TLS alone reproduces or fixes the issue; no other change is needed.

2. The penalty is paid even when cuda_copy carries no traffic. In our runtime, cuda_copy is listed because we register a VRAM region for RDMA communication. However in our benchmark execution path, the real KV Cache transfer path is within one process by calling cudaMemcpyAsync and cudaMemcpy2DAsync API.

   • For KV Cache Offload: We firstly call cudaMemcpy2DAsync to gather the per-layer KV Cache from Paged Attention VRAM region to the VRAM middle buffer region (which is registered for RDMA communication), then we call cudaMemcpyAsync to transfer the KV Cache to lower DRAM tier (the whole DRAM tier region is registered for RDMA communication).
   • For KV Cache Onload: Similar to Offload but just reverse, we first call cudaMemcpyAsync to transfer the KV Cache from DRAM to VRAM middle buffer, then call cudaMemcpy2DAsync to scatter the all-layers KV Cache to the per-layer Paged Attention VRAM region.

Questions for the maintainers

1. What global side effects does cuda_copy MD open (or component load) have on the CUDA context or driver state that could affect the performance of unrelated CUDA work in the same process? We see the regression even when cuda_copy is never selected for a transfer.

2. Is there a supported way to keep cuda_copy in UCX_TLS (as a fallback) without the throughput penalty, or is excluding it via UCX_TLS=^cuda_copy the intended workaround? We'd prefer to keep the fallback available.

3. Are there other transports (e.g. cuda_ipc) that apply similar process-global side effects on MD open that we should be aware of?

4. Is there any guidance on which transports are safe to keep in UCX_TLS for a latency-sensitive, high-concurrency CUDA application that relies on async copy/kernel overlap?

Reproducer sketch

Since the benchmark is based on our in-house inference engine, it's hard to provide a full reproducer script, but the minimal reproduction steps are:

1. Build UCX with CUDA support.
2. Construct nixlAgent with the ucx backend in-process.
3. Start a CUDA application that relies on async cudaMemcpyAsync overlap for throughput (in our case, an LLM inference server with one thread dedicated to inference and another to request preprocessing, which drives data offloading and onloading).
4. Run the workload twice, toggling only UCX_TLS:
   • UCX_TLS=ib,tcp,self,sm (cuda_copy excluded) — baseline QPS.
   • UCX_TLS left default, or cuda_copy explicitly allowed — QPS drops by ~18% (from 2952 to 2411 req/s).

Happy to provide logs, ucx_info output, or any other information required.

[yosefe]

hi @UnpureRationalist , if you are able to patch the UCX code, can you pls try commenting out the call to set SYNC_MEMOPS:

ucx/src/uct/cuda/cuda_copy/cuda_copy_md.c

Line 319 in 7009d7a

UCT_CUDADRV_FUNC(cuda_cuCtxSetFlags_func(CU_CTX_SYNC_MEMOPS),

[brminich]

Also, could you please check if the performance of the cuda_copy run improves when the UCX_SELECT_DISTANCE_MD="" env var is set?

[UnpureRationalist]

Hi @yosefe @brminich , thanks for your suggestions! We tried both, and while neither restored QPS on its own, they helped us narrow down the root cause. We now suspect the regression originates from the UCM CUDA memory hook infrastructure being installed process-globally as a side effect of cuda_copy MD declaring detect_mem_types. We'd appreciate the maintainers' confirmation on whether this is the intended behavior and whether the workaround we found is safe for production.

Response to suggestion 1: commenting out CU_CTX_SYNC_MEMOPS

We tried commenting out the UCT_CUDADRV_FUNC(cuda_cuCtxSetFlags_func(CU_CTX_SYNC_MEMOPS), ...) call in cuda_copy_md.c (line 319), but the QPS regression persisted. This led us to rule out CU_CTX_SYNC_MEMOPS (and the downstream CU_POINTER_ATTRIBUTE_SYNC_MEMOPS per-pointer sync, which is gated behind the same code path) as the root cause.

Response to suggestion 2: UCX_SELECT_DISTANCE_MD=""

We tried setting UCX_SELECT_DISTANCE_MD="" with cuda_copy still in UCX_TLS, but QPS did not recover. From our reading, this variable controls transport selection scoring (ucp_worker_iface_set_sys_device_distance, ucp/core/ucp_worker.c:1225-1243), not UCM hook installation. Setting it to "" means no MD name matches, so all ifaces keep ucs_topo_default_distance — but the hooks are still installed. We'd appreciate confirmation if this understanding is correct.

Our suspected root cause chain

With the help of Claude, we read through the UCX source code and traced the following chain from ucp_init to the hooks landing on every cudaMalloc/cudaFree. We also verified our hypothesis experimentally via the two patches/workarounds described below. However, we are not fully certain this is the intended behavior and would like to ask the maintainers to confirm.

1. cuda_copy MD declares detect_mem_types (uct/cuda/cuda_copy/cuda_copy_md.c:131-132):

   md_attr->detect_mem_types = UCS_BIT(UCS_MEMORY_TYPE_CUDA) |
                               UCS_BIT(UCS_MEMORY_TYPE_CUDA_MANAGED);

   This is the only CUDA transport that sets detect_mem_types. cuda_ipc does not.

2. UCP context collects it (ucp/core/ucp_context.c:1670-1674): during ucp_init, any MD with detect_mem_types is added to context->mem_type_detect_mds[], making num_mem_type_detect_mds > 0.

3. First ucp_mem_map triggers memtype cache lazy creation: when nixl calls ucp_mem_map without specifying memory_type, UCP calls ucp_memory_detect (ucp_context.h:678-715). Because num_mem_type_detect_mds > 0, it takes the slow path and calls ucs_memtype_cache_lookup, which lazily creates the global memtype cache instance.

4. Memtype cache constructor registers UCM event handler (ucs/memory/memtype_cache.c:391-395):

   ucm_set_event_handler(UCM_EVENT_MEM_TYPE_ALLOC |
                         UCM_EVENT_MEM_TYPE_FREE  |
                         UCM_EVENT_FLAG_EXISTING_ALLOC,
                         1000, ucs_memtype_cache_event_callback, self);

5. ucm_set_event_handler installs BISTRO + RELOC hooks on ALL CUDA alloc/free functions (ucm/cuda/cudamem.c): ucm_event_install → UCS_MODULE_FRAMEWORK_LOAD(ucm) (dlopen libucm_cuda.so) → ucm_cudamem_install → ucm_cuda_install_hooks patches every cudaMalloc, cudaFree, cudaMallocAsync, cudaFreeAsync, cudaMallocManaged, cuMemAlloc, cuMemFree, cuMemFreeAsync, etc. via BISTRO (binary patch at function entry) and RELOC (GOT/PLT redirection).

Per-call overhead we suspect is being paid after hooks are installed

Our reading of the code suggests that every hooked CUDA alloc/free in the entire process would go through:

ucm_event_enter()                        // pthread_rwlock_rdlock(&ucm_event_lock)
  → original CUDA alloc/free call
  → ucm_event_dispatch(MEM_TYPE_ALLOC|FREE)
      → ucs_memtype_cache_event_callback()
          → ucs_memtype_cache_update_internal()
              → ucs_spin_lock(&memtype_cache->lock)
              → ucs_pgtable_search_range()  // page table search
              → ucs_pgtable_remove()        // remove overlapping region
              → ucs_memtype_cache_insert()  // insert new region
              → ucs_spin_unlock(&memtype_cache->lock)
ucm_event_leave()                        // pthread_rwlock_unlock(&ucm_event_lock)

For cudaFree/cudaFreeAsync/cudaFreeHost, there is an additional cuMemGetAddressRange() driver call inside ucm_cuda_dispatch_mem_free (because the size parameter is 0 for these wrappers), extending the lock-hold duration.

In our inference engine, the KV Cache offload/onload path calls cudaMemcpyAsync/cudaMemcpy2DAsync frequently, and the inference engine itself calls cudaMallocAsync/cudaFreeAsync for temporary buffers. We suspect that every one of these calls pays the rwlock + spinlock + page table overhead, which would compress the async-copy/kernel overlap window and degrade QPS by ~18%. We'd appreciate confirmation on whether this overhead profile matches what the maintainers would expect.

Verification: early return in ucm_cudamem_install appears to restore QPS

As a sanity check, we tried adding an early return at the top of ucm_cudamem_install in ucx/src/ucm/cuda/cudamem.c, which prevents the BISTRO/RELOC hooks from being installed while still allowing cuda_copy MD to open and detect_mem_types to be declared:

static ucs_status_t ucm_cudamem_install(int events)
{
    static pthread_mutex_t install_mutex = PTHREAD_MUTEX_INITIALIZER;
    static int driver_api_hooks          = 0;
    static int runtime_api_hooks         = 0;
    ucs_status_t status                  = UCS_OK;

    printf("ucm_cudamem_install early return\n");

    return status;   // ← hooks never installed, QPS restored to baseline
    // ... rest of function unreachable ...
}

With this patch and cuda_copy still in UCX_TLS, QPS returned to the baseline level (matching the UCX_TLS=ib,tcp,self,sm run) in our tests. We take this as strong evidence that:

• The detect_mem_types declaration itself is harmless.
• The UCM CUDA memory hooks are the likely cause of the QPS regression.

We'd appreciate the maintainers' perspective on whether this conclusion is correct.

Verification: UCX_MEMTYPE_CACHE=no also appears to restore QPS

As a second data point, setting UCX_MEMTYPE_CACHE=no with cuda_copy still in UCX_TLS also restored QPS to baseline in our tests:

export UCX_TLS=ib,tcp,self,sm,cuda_copy
export UCX_MEMTYPE_CACHE=no
# benchmark invocation — QPS matches baseline

UCX_MEMTYPE_CACHE=no causes ucs_memtype_cache_get_global() (ucs/memory/memtype_cache.c:59) to return NULL without creating the cache, so the ucm_set_event_handler call inside the cache constructor (memtype_cache.c:391-395) is never reached and the UCM hooks are never installed. The trade-off is that num_mem_type_detect_mds is still > 0, so each ucp_mem_map without an explicit memory type falls through to ucp_memory_detect_slowpath, which calls uct_md_mem_query on every mem_type_detect_md (ucp_context.c:2589-2591). This adds a per-registration CUDA driver pointer-attribute query but avoids the per-alloc/free hook overhead — a favorable trade-off for our workload.

Questions for the maintainers

1. Could the maintainers confirm whether the UCM CUDA memory hooks being installed process-globally as a side effect of cuda_copy MD declaring detect_mem_types is the intended behavior? If so, is there a recommended way to keep cuda_copy in UCX_TLS (as a fallback) without triggering the hooks — or is UCX_MEMTYPE_CACHE=no the intended workaround?

2. Is UCX_MEMTYPE_CACHE=no safe for production use when cuda_copy is kept in UCX_TLS? Are there any correctness concerns with relying on ucp_memory_detect_slowpath (per-registration uct_md_mem_query) instead of the memtype cache?

3. Are there other transports (e.g. cuda_ipc) that apply similar process-global side effects on MD open that we should be aware of?