Skip to content

Conversation

@ypapadop-amd
Copy link
Contributor

@ypapadop-amd ypapadop-amd commented Jan 6, 2026

Motivation

The use of the HSA vmem API with XDNA devices was not possible due to its dependence on the amdgpu driver API use in runtime.cpp. This PR moves the relevant calls in the KFDDriver and simplifies the Driver API that Runtime::VMemoryHandleMap uses.

Technical Details

  1. A new function Driver::CreateShareableHandle was added that creates handles for vmem-related allocations.
  2. Destruction of handles is split because for amdgpu it appears they are ref-counted wheareas for xdna-driver they are not and that causes the same BO to be destroyed more than once (see DestroyImportedShareableHandle vs DestroyShareableHandle).
  3. Inaccessible pages are created to get a virtual address, since ThunkHandle is a only a pointer. This wastes an extra page for each XDNADriver-based allocation if not needed to be mapped at allocation, but it guarantees the address uniqueness among drivers.

Follow-ups:

  1. It seems that we ended up with two different handle concepts merged in one (core::ShareableHandle). They may need to be separated to two, one handle that does not own an allocation (core::ShareableHandle) and one that does (core::MemoryHandle); the latter is probably similar to Runtime::MemoryHandle. Further discussion is welcome.
  2. The ThunkHandle definition creates the need to waste a page in XDNADriver although we already have a BO handle. It's unclear how to address that without an extra mapping, e.g., ThunkHandle -> {driver, handle}.
  3. The fn_amdgpu_device_get_fd is still set by Runtime. It's unclear if this needs to move in KFDDriver or not.

JIRA ID

NA

Test Plan

  • rocrtst runs
  • Local testing via AIE-specific tests.

Test Result

Successful run.

Submission Checklist

@ypapadop-amd ypapadop-amd self-assigned this Jan 6, 2026
Copilot AI review requested due to automatic review settings January 6, 2026 21:30
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors the HSA vmem API to remove the dependency on the KFD driver, enabling support for XDNA devices. The main change consolidates driver-specific operations into their respective driver implementations rather than having them in the generic Runtime class.

Key changes:

  • Introduces Driver::CreateShareableHandle to encapsulate driver-specific handle creation logic previously in Runtime::GetAmdgpuDeviceArgs
  • Splits handle destruction into DestroyShareableHandle and DestroyImportedShareableHandle to account for different reference counting behaviors between drivers
  • Updates XDNADriver to create anonymous mappings for memory-only allocations to ensure unique virtual addresses

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
runtime.cpp Removes GetAmdgpuDeviceArgs and refactors vmem mapping to use new driver API
runtime.h Adds GetAmdgpuRenderFdFunc accessor and removes GetAmdgpuDeviceArgs declaration
driver.h Defines new CreateShareableHandle, DestroyShareableHandle, and DestroyImportedShareableHandle APIs
amd_xdna_driver.h Updates driver interface signatures and adds unmap_vaddr field to BOHandle
amd_virtio_driver.h Updates driver interface signatures
amd_kfd_driver.h Updates driver interface signatures
amd_xdna_driver.cpp Implements new driver APIs and refactors memory allocation to support memory-only mode
amd_kfd_virtio_driver.cpp Implements stub functions for new driver APIs
amd_kfd_driver.cpp Moves vmem-related logic from Runtime into KfdDriver implementation

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@ypapadop-amd ypapadop-amd force-pushed the users/ypapadop-amd/xdna-vmem-support branch 2 times, most recently from 327b968 to 843330b Compare January 7, 2026 15:31
MemoryHandle *memHandle = mappedHandle->mem_handle;

// Export memory from owner agent.
hsa_status_t status = memHandle->agentOwner()->driver().ExportDMABuf(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can these calls to ExportDMABuf/ImportDMABuf be replaced with CreateShareableHandle?

Copy link
Contributor Author

@ypapadop-amd ypapadop-amd Jan 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the general case no, since there are potentially different source and target drivers involved.

It may be worth it as an optimization, i.e., source and target driver is the same. I'm not entirely sure.

@ypapadop-amd ypapadop-amd force-pushed the users/ypapadop-amd/xdna-vmem-support branch 2 times, most recently from b41af03 to d97a7a1 Compare January 9, 2026 15:30
@ypapadop-amd ypapadop-amd force-pushed the users/ypapadop-amd/xdna-vmem-support branch 3 times, most recently from da1c503 to 381e551 Compare January 13, 2026 15:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants