fix(libfabric): Use PCI bus ID for GPU-to-EFA mapping #1149
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
akkart-aws
requested review from
amitrad-aws,
fengjica and
yexiang-aws
as code owners
|
👋 Hi akkart-aws! Thank you for contributing to ai-dynamo/nixl. Your PR reviewers will review your contribution then trigger the CI to test your changes. 🚀 |
akkart-aws
force-pushed
the
fix_gpu_id
branch
from
8ccd2ba to
3ba21b8
Compare
fengjica
previously approved these changes
Dec 23, 2025
akkart-aws
force-pushed
the
fix_gpu_id
branch
from
3ba21b8 to
277c6c4
Compare
amitrad-aws
previously approved these changes
Dec 24, 2025
fengjica
previously approved these changes
Dec 26, 2025
akkart-aws
dismissed stale reviews from fengjica and amitrad-aws
via
ab61957
akkart-aws
force-pushed
the
fix_gpu_id
branch
from
277c6c4 to
ab61957
Compare
fengjica
previously approved these changes
Dec 31, 2025
Contributor
|
/build |
Contributor
|
/ok to test ab61957 |
akkart-aws
force-pushed
the
fix_gpu_id
branch
2 times, most recently
from
6bffe05 to
f4688e2
Compare
akkart-aws
force-pushed
the
fix_gpu_id
branch
2 times, most recently
from
3fe6bfe to
09ddd26
Compare
Contributor
|
/build |
Contributor
|
/ok to test 3e21e6a |
akkart-aws
force-pushed
the
fix_gpu_id
branch
3 times, most recently
from
93d0b2f to
25b96a9
Compare
Fix incorrect EFA device selection when CUDA_VISIBLE_DEVICES is set by using PCI bus IDs instead of enumeration order. Query physical GPU via cuPointerGetAttributes(), map to hwloc topology index, and select correct EFA devices based on PCIe proximity. Fixes GPU device ID mismatch between CUDA and hwloc enumeration that caused wrong EFA rails to be selected in vLLM and multi-GPU workloads.
Contributor
|
/build |
Contributor
|
/ok to test c596b28 |
fengjica
approved these changes
Jan 7, 2026
Contributor
|
/build |
Contributor
|
/ok to test c596b28 |
Contributor
|
/build |
Contributor
|
/ok to test c596b28 |
ovidiusm
pushed a commit
to ovidiusm/nixl
that referenced
this pull request
Jan 8, 2026
mcuiaws
pushed a commit
to mcuiaws/nixl
that referenced
this pull request
Jan 8, 2026
(cherry picked from commit 7233020)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What?
Fix incorrect EFA device selection when using
CUDA_VISIBLE_DEVICESby switching from GPU ID-based to PCI bus ID-based topology mapping.Why?
The previous implementation used GPU IDs (0, 1, 2...) to map GPUs to EFA devices. This broke when
CUDA_VISIBLE_DEVICESwas set because:CUDA_VISIBLE_DEVICES=1makes physical GPU 1 appear as device 0 to the applicationExample failure case:
How?
Changed topology mapping from GPU ID to GPU-PCI bus ID:
gpu_to_efa_devices→pci_to_efa_devicesgetEfaDevicesForGpu(int)→getEfaDevicesForGPUPci(string)Backend queries PCI bus ID from memory address:
cudaQueryAddr()to return GPU'S PCI bus ID viacuDeviceGetPCIBusId()registerMem()Rail manager uses PCI bus ID for rail selection:
registerMemory()andselectRailsForMemory()to accept PCI bus IDtopology->getEfaDevicesForGPUPci(pci_bus_id)instead of GPU ID lookupBenefits:
CUDA_VISIBLE_DEVICES- GPU-PCI bus ID is stable regardless of device renumbering