Fix issues related to GPUNetIO #1112
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
👋 Hi foraxe! Thank you for contributing to ai-dynamo/nixl. Your PR reviewers will review your contribution then trigger the CI to test your changes. 🚀 |
brminich
reviewed
Jan 8, 2026
Comment on lines
+473
to
+481
| if (env_dbg) { | ||
| std::ostringstream oss; | ||
| oss << "[dbg] MR path: " << (dmabuf_fd >= 0 ? "dmabuf" : "peermem/ibv_reg_mr") | ||
| << ", addr 0x" << std::hex << std::uppercase << (uintptr_t)addr << std::dec | ||
| << " len 0x" << std::hex << (uint64_t)tot_size << std::dec | ||
| << " lkey 0x" << std::hex << (uint32_t)lkey << std::dec | ||
| << " rkey 0x" << std::hex << (uint32_t)rkey << std::dec; | ||
| NIXL_INFO << oss.str(); | ||
| } |
Contributor
There was a problem hiding this comment.
pls use NIXL_DEBUG instead of this
Comment on lines
+466
to
+467
| // lkey = htobe32(ibmr->lkey); | ||
| // rkey = htobe32(ibmr->rkey); |
| ((volatile struct docaNotif *)notif_send_cpu)->msg_buf = msg_buf; | ||
| ((volatile struct docaNotif *)notif_send_cpu)->msg_lkey = notif->send_mr->get_lkey(); | ||
| ((volatile struct docaNotif *)notif_send_cpu)->msg_size = newMsg.size(); | ||
| // ((volatile struct docaNotif *)notif_send_cpu)->msg_lkey =notif->send_mr->get_lkey(); |
| xferReqRingCpu[treq->end_pos - 1].lbuf_notif = notif_addr; | ||
| xferReqRingCpu[treq->end_pos - 1].lkey_notif = notif->send_mr->get_lkey(); | ||
| uint32_t notif_lkey_host = notif->send_mr->get_lkey(); | ||
| // xferReqRingCpu[treq->end_pos - 1].lkey_notif = notif_lkey_host; |
Comment on lines
+1260
to
+1261
| //uint32_t lkey = xferReqRingCpu[pos].lkey[idx]; | ||
| //uint32_t rkey = xferReqRingCpu[pos].rkey[idx]; |
Comment on lines
+1077
to
+1088
| NIXL_INFO << "GPUNETIO registerMem publish dev " << priv->devId << " addr " | ||
| << format_hex((uintptr_t)priv->mr->get_addr()) << " len " | ||
| << format_hex((uint64_t)priv->mr->get_tot_size()) << " lkey " | ||
| << format_hex(lkey) << " rkey " << format_hex(rkey); | ||
| if (debug_dump) { | ||
| std::ostringstream oss; | ||
| oss << " [dbg] publish raw: dev=" << priv->devId << " addr_dec=" | ||
| << (uintptr_t)priv->mr->get_addr() << " len_dec=" | ||
| << (uint64_t)priv->mr->get_tot_size() << " lkey_dec=" << lkey << " rkey_dec=" | ||
| << rkey; | ||
| NIXL_INFO << oss.str(); | ||
| } |
Contributor
There was a problem hiding this comment.
imo it should be NIXL_DEBUG
| ss << (int)ipv4_addr[0] << "." << (int)ipv4_addr[1] << "." << (int)ipv4_addr[2] << "." | ||
| << (int)ipv4_addr[3]; | ||
| str = ss.str(); | ||
| std::cout << "getConnInfo DOCA: " << str << std::endl; |
Contributor
There was a problem hiding this comment.
we should not print anything by default except errors
pls use NIXL_DEBUG
Comment on lines
+559
to
+561
| // server_addr.sin_addr.s_addr = INADDR_ANY; | ||
| /* listen on any interface */ | ||
| std::memcpy(&server_addr.sin_addr.s_addr, ipv4_addr, sizeof(ipv4_addr)); |
Contributor
There was a problem hiding this comment.
Suggested change
| // server_addr.sin_addr.s_addr = INADDR_ANY; | |
| /* listen on any interface */ | |
| std::memcpy(&server_addr.sin_addr.s_addr, ipv4_addr, sizeof(ipv4_addr)); | |
| std::memcpy(&server_addr.sin_addr.s_addr, ipv4_addr, sizeof(ipv4_addr)); |
| notifMap[remote_agent] = notif; | ||
| ((volatile struct docaNotif *)notif_fill_cpu)->msg_buf = (uintptr_t)notif->recv_addr; | ||
| ((volatile struct docaNotif *)notif_fill_cpu)->msg_lkey = notif->recv_mr->get_lkey(); | ||
| // ((volatile struct docaNotif *)notif_fill_cpu)->msg_lkey = notif->recv_mr->get_lkey(); |
Contributor
There was a problem hiding this comment.
Suggested change
| // ((volatile struct docaNotif *)notif_fill_cpu)->msg_lkey = notif->recv_mr->get_lkey(); |
| else | ||
| swap_keys_config = true; | ||
| } | ||
| const char *env_dbg = std::getenv("NIXL_GPUNETIO_DEBUG_DUMP"); |
Contributor
There was a problem hiding this comment.
why not using NIXL_DEBUG instead?
Contributor
|
@e-ago, can you pls review? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What?
This PR fixs the GPUNetIO backend to make nixlbench run on intranode.
Why?
The DOCA couldn't start transfer data in our server envs, with following log:
prepare_write radd 7f8637fff010 rkey 130100 ladd 7f6180000000 lkey d2100100 size 4096 prepare_write radd 7f8637fff010 rkey 130100 ladd 7f6180000000 lkey d2100100 size 4096 >>>>>>> CUDA rdma write kernel pos 0 posted 512 buffers from base_wqe_idx 0 got completion with err: syndrome=0x5, vendor_err_synd=0xf9, hw_err_synd=0, hw_synd_type=0, wqe_counter=65281 wqe_qpn=ea030008 kernel_progress: block 0 error CQE! poll_status -5 wqe 511 index 0I had tried many versions of code in community, but ended up with above log.
So, i am uploading an working version which ran on our local servers.
Ref:
#952 (comment)
#788
NVIDIA-DOCA/gpunetio#2 (comment)
How to run:
NIXL_DOCASIM_IPV4_OVERRIDEuse the inet ip of mlx5_bond0. One can obtain it by running commandip acc @e-ago @ovidiusm