Add resident device change call #1517

lslusarczyk · 2025-09-11T12:30:55Z

UMF part of https://github.com/intel/llvm/blob/sycl/sycl/doc/extensions/supported/sycl_ext_oneapi_peer_access.asciidoc feature.

UR/llvm part is in intel/llvm#19257

include/umf/memory_pool_ops.h

src/libumf.map

bratpiorka · 2025-09-11T12:39:25Z

src/memory_provider.c

+        hProvider->provider_priv, deviceIndex, isAdding);
+    checkErrorAndSetLastProvider(res, hProvider);
+    return res;
+}


add empty line

src/provider/provider_level_zero.c

bratpiorka · 2025-09-11T12:48:49Z

src/provider/provider_cuda.c

@@ -802,6 +802,15 @@ static umf_result_t cu_memory_provider_get_allocation_properties_size(
    return UMF_RESULT_ERROR_INVALID_ARGUMENT;
 }

+static umf_result_t


add a "default" handler in memory_provider.c (see umfDefaultCtlHandle) and do not modify CUDA provider

Where exactly in the code you think "default" is missing?

do not modify CUDA provid

Rejected. umf_memory_provider_ops_t type received a new field and it should be somehow initialized in UMF_CUDA_MEMORY_PROVIDER_OPS as all other fields. Please explain if you still want me to remove this change.

Please initialize cuda ops .ext_resident_device_change to NULL.
Next, define the umfDefaultResidentDeviceChange in src/memory_provider.c that returns UMF_RESULT_ERROR_NOT_SUPPORTED and set this handler for each created provider if the ext_resident_device_change is set to NULL (see how this is done for other default handlers like "umfDefaultCloseIPCHandle")

ok, applied

bratpiorka · 2025-09-11T12:51:31Z

src/provider/provider_tracking.c

@@ -1336,6 +1336,15 @@ static umf_result_t trackingGetAllocationPropertiesSize(
        p->hUpstream, memory_property_id, size);
 }

+static umf_result_t trackingResidentDeviceChange(void *provider,


Rejected. umf_memory_provider_ops_t type received a new field and it should be somehow initialized in UMF_TRACKING_MEMORY_PROVIDER_OPS as all other fields. Please explain if you still want me to remove this change.

applied in the same way as in cuda provider

bratpiorka · 2025-09-11T12:51:53Z

test/common/provider_null.c

@@ -162,6 +162,15 @@ static umf_result_t nullGetAllocationPropertiesSize(
    return UMF_RESULT_SUCCESS;
 }

+static umf_result_t nullResidentDeviceChange(void *provider,


rejected, same reason as above

bratpiorka · 2025-09-12T08:15:51Z

src/provider/provider_level_zero.c

    if (!hParams) {
        LOG_ERR("Level Zero memory provider params handle is NULL");
        return UMF_RESULT_ERROR_INVALID_ARGUMENT;
    }

-    if (deviceCount && !hDevices) {
-        LOG_ERR("Resident devices array is NULL, but deviceCount is not zero");
+    if (residentDevicesCount && !residentDevicesIndices) {


please also check if residentDevicesCount == 0 but residentDevicesIndices != NULL

additionally, should indices be unique?

ok, added checking for uniqueness

bratpiorka · 2025-09-12T08:18:39Z

src/provider/provider_level_zero.c

@@ -930,6 +987,155 @@ static umf_result_t ze_memory_provider_get_allocation_properties_size(
    return UMF_RESULT_ERROR_INVALID_ARGUMENT;
 }

+struct ze_memory_provider_resident_device_change_data {
+    bool isAdding;


bratpiorka · 2025-09-12T08:19:39Z

src/provider/provider_level_zero.c

+        change_data->source_memory_provider
+            ->device_handles[change_data->peer_device_index];
+
+    // TODO: add assertions to UMF and change it to be an assertion


use ASSERT() macros from src/utils/utils_assert.h

Postponed to some future PR. I need here a permanent ASSERT, not a one that compiles on in debug config like ASSERT does.

you assert only things which are can be an artefact of some internal umf error - if it is true debug only assert is fine. If an error can be a result of an user input you just handle the issue and return a error to the user

bratpiorka · 2025-09-12T08:21:28Z

src/provider/provider_level_zero.c

+            info->props.base, info->props.base_size);
+    } else {
+        result = ZE_RESULT_SUCCESS;
+        // TODO: currently not implemented call evict here


do you plan to add the missing code here?

Not in this PR.

bratpiorka · 2025-09-12T08:24:18Z

src/provider/provider_level_zero.c

+    if (ze_provider->resident_device_count == 0 ||
+        existing_peer_index == ze_provider->resident_device_count) {
+        // not found
+        if (!isAdding) { // impossible for UR, should be an assertion


if this should be an assertion, please use the ASSERT() macro from src/utils/utils_assert.h
Also, move the comment to the next line

To be changed in future PR when umf will have permanent assertions. Comments moved.

bratpiorka · 2025-09-12T08:24:27Z

src/provider/provider_level_zero.c

+
+static umf_result_t
+ze_memory_provider_resident_device_change(void *provider, uint32_t device_index,
+                                          bool isAdding) {


bratpiorka · 2025-09-12T08:24:46Z

src/provider/provider_level_zero.c

+        // adding case
+        if (ze_provider->device_count <=
+            ze_provider
+                ->resident_device_count) { // impossible for UR, should be an assertion


if this should be an assertion, please use the ASSERT() macro from src/utils/utils_assert.h
Also, move the comment to the next line

postponed when UMF have permanent assertions, comments moved

bratpiorka · 2025-09-12T08:24:54Z

src/provider/provider_level_zero.c

+        }
+
+        if (ze_provider->device_count <=
+            device_index) { // impossible for UR, should be an assertion


applied as above

bratpiorka · 2025-09-12T08:25:32Z

src/provider/provider_level_zero.c

+
+    } else {
+        // found
+        if (isAdding) { // impossible for UR, should be an assertion


applied as above

bratpiorka · 2025-09-12T08:27:28Z

src/provider/provider_level_zero.c

+        LOG_ERR("umfMemoryTrackerIterateAll did not manage to do some change "
+                "numFailed:%d, numSuccess:%d",
+                privData.success_changes, privData.failed_changes);
+        return UMF_RESULT_ERROR_INVALID_ARGUMENT; // probably some other result is better, best just change into assertion


you could use UMF_RESULT_ERROR_MEMORY_PROVIDER_SPECIFIC
or change to assert

src/critnib/critnib.c

lukaszstolarczuk · 2025-09-15T09:02:40Z

src/libumf.map

@@ -54,6 +54,7 @@ UMF_1.0 {
        umfMemoryProviderPurgeForce;
        umfMemoryProviderPurgeLazy;
        umfMemoryProviderPutIPCHandle;
+        umfMemoryProviderResidentDeviceChange;


pls move down in the file (after a comment # Added in UMF_1.0)

actually, mea culpa, pls make a new sction, we're already past v1.0 - we should have now # Added in UMF_1.1 section

lukaszstolarczuk · 2025-09-15T09:02:58Z

src/libumf.def

@@ -60,6 +60,7 @@ EXPORTS
    umfMemoryProviderPurgeForce
    umfMemoryProviderPurgeLazy
    umfMemoryProviderPutIPCHandle
+    umfMemoryProviderResidentDeviceChange


pls move down in the file (after a comment ; Added in UMF_1.0)

first of all, wrong function was moved - umfMemoryProviderResidentDeviceChange is the new one

mea culpa, it should be already in ; Added in UMF_1.1 section

Moved down in 01d74ba

lukaszstolarczuk · 2025-09-15T09:06:07Z

src/memory_provider.c

@@ -606,3 +619,16 @@ umf_result_t umfMemoryProviderGetAllocationPropertiesSize(
    checkErrorAndSetLastProvider(res, hProvider);
    return res;
 }
+
+umf_result_t
+umfMemoryProviderResidentDeviceChange(umf_memory_provider_handle_t hProvider,


do we have any tests for this function? including negative tests

no, success path will be tested by UR tests
failure path does not happen as UR is the only one using this functionality and it never issues wrong parameters

If you think some test case is important and possible to test within UMF please write which case and please guide me where to add it and what other test to use as example.

UR is an separate project - we need at least few simple test cases to do not decrease code coverage.

ok, I've added some tests in 88f672b

still missing tests for new functions, even simple negative test would be better than nothing (you can put them in generic API tests: test/memoryProviderAPI.cpp)

OK, applied, In 53ec753 I've added UTs covering all new code paths.

lplewa · 2025-09-15T10:18:31Z

include/umf/memory_provider_ops.h

+    /// @brief Adds or removes devices on which allocations should be made
+    ///        resident.
+    /// @param provider handle to the memory provider
+    /// @param device_index identifier of device
+    /// @param is_adding Boolean indicating if peer is to be removed or added
+    /// @return UMF_RESULT_SUCCESS on success or appropriate error code on
+    ///         failure.
+    umf_result_t (*ext_resident_device_change)(void *provider,
+                                               uint32_t device_index,
+                                               bool is_adding);
+


This function should not be in OPS. This is something very specific to L0 provider. Resident devices are passed to L0 provider thru provider specific params, so control of them should be olso be done through provider specific API, this is why i think with should be implemented through CTL

Rejected. We discussed this in teams (4th Aug, 25) we do it as ops (Piotr, sorry in Polish): co do tego API co Ty robisz (w stylu daj mi wszystkie zaalokowane page'e) ja bym chyba sugerował zrobić API a nie robić przez CTLa, Rafał: no to faktycznie chyba dedykowane API lepiej pasuje niż CTL.

CTL should be for statistics, not a universal, hard-to-read tool to implement any API, Łukasz (me): Być może będziecie chcieli uprościć ext_ctl by nie był maszynką, którą mozna zaimplementować wszystko, a jedynie służył do statystyk. Ale to już ja się na tym nie znam - zostawiam do przemyśleń.

We talked about API to iterate over all allocations, not about API to modify some internal settings of the L0

I can see it both ways. My comment during that discussion was about a generic functionality implementable for all providers. This isn't that.
I also remember making a point about CTL being useful for provider/pool-specific functionality.

On the other hand, adding an API function is simpler.

On the other hand, adding an API function is simpler.

Thank you for the comment. Chosen simpler way.

lplewa · 2025-09-15T10:18:33Z

include/umf/providers/provider_level_zero.h

 /// @brief  Set the resident devices in the parameters struct.
 /// @param  hParams handle to the parameters of the Level Zero Memory Provider.
-/// @param  hDevices array of devices for which the memory should be made resident.
-/// @param  deviceCount number of devices for which the memory should be made resident.
+/// @param  hDevices array of all devices for which the memory can be made resident.
+/// @param  deviceCount number of devices for which the memory can be made resident.
+/// @param  residentDevicesIndices array of indices in all devices array to devices for which the memory should be made resident.
+/// @param  residentDevicesCount number of items in indices array.
 /// @return UMF_RESULT_SUCCESS on success or appropriate error code on failure.
 umf_result_t umfLevelZeroMemoryProviderParamsSetResidentDevices(
    umf_level_zero_memory_provider_params_handle_t hParams,
-    ze_device_handle_t *hDevices, uint32_t deviceCount);
+    ze_device_handle_t *hDevices, uint32_t deviceCount,
+    uint32_t *residentDevicesIndices, uint32_t residentDevicesCount);


This is an API(and ABI) break - we are post 1.0 release so you cannot do changes like this.

Shall I bump some number? Only UR uses this API and I am changing UR right now as well.

Even if UR uses only this API, old UR should work with new umf. And we will not do 2.0 release just after 1.0.

You cannot brake API. If changes are needed we have to keep old function working correctly and add new one.

We can't break backwards compatibility with SYCL / UR. Changing a major version is a significant undertaking that involves updating the components across all the different layers (UMF is nearly the lowest-most component in the stack). We've been burned on this in the past, and it's very disruptive.

In this case, my suggestion would be to simply create a new function that sets the indices.

OK, applied, I rewrote code in 53ec753 to be backward compatible

ldorau

14 / 16 files reviewed (will be continued)

pbalcer · 2025-09-16T08:54:51Z

include/umf/memory_provider_ops.h

+    ///         failure.
+    umf_result_t (*ext_resident_device_change)(void *provider,
+                                               uint32_t device_index,
+                                               bool is_adding);


I'd rather we have two functions - add/remove.

I will not apply this, please, because this will introduce too much copy-pasted code.

lplewa · 2025-09-19T13:16:18Z

I still oppose against adding provider specific code to ops interface.
If you don't want to do it thru CTL just implement it as umfLevelZeroMemoryProviderResidentDeviceChange() in provider_level_zero.h which do not go throught OPS. This function will act as an "level zero specific api extension".

note: remember that you will have to call umfMemoryProviderGetPriv() in this function to retrieve ze_memory_provider_t from provider handle

bratpiorka · 2025-09-19T13:23:54Z

src/libumf.def

@@ -19,6 +19,7 @@ EXPORTS
    umfCUDAMemoryProviderParamsSetContext
    umfCUDAMemoryProviderParamsSetDevice
    umfCUDAMemoryProviderParamsSetMemoryType
+    umfLevelZeroMemoryProviderParamsSetResidentDevices


please keep alphabetical order

bratpiorka · 2025-09-19T13:25:00Z

src/provider/provider_level_zero.c

@@ -174,11 +177,18 @@ static ze_memory_type_t umf2ze_memory_type(umf_usm_memory_type_t memory_type) {
 }

 static void init_ze_global_state(void) {
+
+    char *lib_name = getenv("UMF_ZE_LOADER_LIB_NAME");


why we need this code? will this code path be used by UR? if not please remove this

lplewa · 2025-09-19T13:21:25Z

src/provider/provider_level_zero.c

@@ -174,11 +177,18 @@ static ze_memory_type_t umf2ze_memory_type(umf_usm_memory_type_t memory_type) {
 }

 static void init_ze_global_state(void) {
+
+    char *lib_name = getenv("UMF_ZE_LOADER_LIB_NAME");


Can you do this thru CTL please.

We created CTL to limit new envvariables.

lplewa · 2025-09-19T13:26:46Z

src/provider/provider_level_zero.c

    hParams->resident_device_handles = hDevices;
    hParams->resident_device_count = deviceCount;



Shouldn't we copy this array? - user after call to this function can do anything with this array.

lplewa · 2025-09-19T13:29:52Z

src/provider/provider_level_zero.c

        ze_result = g_ze_ops.zeContextMakeMemoryResident(
            ze_provider->context, ze_provider->resident_device_handles[i],
            *resultPtr, size);
        if (ze_result != ZE_RESULT_SUCCESS) {
+            utils_read_unlock(&ze_provider->resident_device_rwlock);


If we return an failure then it's means allocation failed, but allocation is still "done". IMHO we should free it here.

lplewa · 2025-09-19T13:33:02Z

src/provider/provider_level_zero.c

+        change_data->source_memory_provider
+            ->device_handles[change_data->peer_device_index];
+
+    // TODO: add assertions to UMF and change it to be an assertion


you assert only things which are can be an artefact of some internal umf error - if it is true debug only assert is fine. If an error can be a result of an user input you just handle the issue and return a error to the user

lplewa · 2025-09-19T13:33:09Z

src/provider/provider_level_zero.c

+
+    // TODO: add assertions to UMF and change it to be an assertion
+    if (info->props.base != (void *)key) {
+        LOG_ERR("key:%p is different than base:%p", (void *)key,


lplewa · 2025-09-19T13:37:40Z

src/utils/utils_log.h

-#define LOG_DEBUG(...) utils_log(LOG_DEBUG, __func__, __VA_ARGS__);
-#define LOG_INFO(...) utils_log(LOG_INFO, __func__, __VA_ARGS__);
-#define LOG_WARN(...) utils_log(LOG_WARNING, __func__, __VA_ARGS__);
-#define LOG_ERR(...) utils_log(LOG_ERROR, __func__, __VA_ARGS__);
-#define LOG_FATAL(...) utils_log(LOG_FATAL, __func__, __VA_ARGS__);
-
-#define LOG_PDEBUG(...) utils_plog(LOG_DEBUG, __func__, __VA_ARGS__);
-#define LOG_PINFO(...) utils_plog(LOG_INFO, __func__, __VA_ARGS__);
-#define LOG_PWARN(...) utils_plog(LOG_WARNING, __func__, __VA_ARGS__);
-#define LOG_PERR(...) utils_plog(LOG_ERROR, __func__, __VA_ARGS__);
-#define LOG_PFATAL(...) utils_plog(LOG_FATAL, __func__, __VA_ARGS__);
+#ifdef UMF_DEVELOPER_MODE
+#define UMF_STRINGIFY(x) #x
+#define UMF_TOSTRING(x) UMF_STRINGIFY(x)
+#define UMF_FUNC_DESC() __FILE__ ":" UMF_TOSTRING(__LINE__)
+#else
+#define UMF_FUNC_DESC() __func__
+#endif
+
+#define LOG_DEBUG(...) utils_log(LOG_DEBUG, UMF_FUNC_DESC(), __VA_ARGS__);
+#define LOG_INFO(...) utils_log(LOG_INFO, UMF_FUNC_DESC(), __VA_ARGS__);
+#define LOG_WARN(...) utils_log(LOG_WARNING, UMF_FUNC_DESC(), __VA_ARGS__);
+#define LOG_ERR(...) utils_log(LOG_ERROR, UMF_FUNC_DESC(), __VA_ARGS__);
+#define LOG_FATAL(...) utils_log(LOG_FATAL, UMF_FUNC_DESC(), __VA_ARGS__);
+
+#define LOG_PDEBUG(...) utils_plog(LOG_DEBUG, UMF_FUNC_DESC(), __VA_ARGS__);
+#define LOG_PINFO(...) utils_plog(LOG_INFO, UMF_FUNC_DESC(), __VA_ARGS__);
+#define LOG_PWARN(...) utils_plog(LOG_WARNING, UMF_FUNC_DESC(), __VA_ARGS__);
+#define LOG_PERR(...) utils_plog(LOG_ERROR, UMF_FUNC_DESC(), __VA_ARGS__);
+#define LOG_PFATAL(...) utils_plog(LOG_FATAL, UMF_FUNC_DESC(), __VA_ARGS__);


This should be a separate PR.

There are comments which i whould make - but i strongly believe that this discussion should happen in the separate PR to do not block this PR

lplewa · 2025-09-19T14:05:12Z

src/provider/provider_level_zero.c

+        // found
+        if (is_adding) {
+            utils_write_unlock(&ze_provider->resident_device_rwlock);
+            // impossible for UR, should be an assertion


imho we should not assume that user will not do something then is should be an assertion. If it comes it for any kind of user it should be validated an correct error should be returned

lplewa · 2025-09-19T14:09:47Z

src/provider/provider_level_zero.c

+
+    uint32_t existing_peer_index = 0;
+    utils_write_lock(&ze_provider->resident_device_rwlock);
+    while (existing_peer_index < ze_provider->resident_device_count &&


NIT: this code whould be much easier to read if it whould be a for loop

for (index = 0; index < count; index++) if (device[index] == index) break;

lplewa · 2025-09-19T14:36:06Z

src/provider/provider_level_zero.c

+            const uint32_t new_capacity =
+                ze_provider->resident_device_capacity * 2 +
+                1; // +1 to work also with old capacity == 0
+            ze_device_handle_t *new_handles =


Do this code is performance critical, or will be called multiple times

If not i would prefer to have this array just always reallocated by one,
If yes can we abstract this vector like structure to separate module, or even better just use list structure with is included in umf.

lplewa · 2025-09-19T14:39:34Z

src/provider/provider_level_zero.c

+        .failed_changes = 0,
+    };
+
+    umf_result_t result = umfMemoryTrackerIterateAll(


This is "hacky" and it will not work if someone uses pool without tracker or just use provider without pool.

We allready solved this issue in other providers were we keep track allocation by the provider (for example please check os_provider)

lslusarczyk requested a review from a team as a code owner September 11, 2025 12:30

bratpiorka requested changes Sep 11, 2025

View reviewed changes

bratpiorka requested review from lplewa and ldorau September 11, 2025 12:53

bratpiorka reviewed Sep 12, 2025

View reviewed changes

lslusarczyk force-pushed the memresident branch from d53a23f to 9d7e12c Compare September 15, 2025 08:16

lukaszstolarczuk reviewed Sep 15, 2025

View reviewed changes

lplewa requested changes Sep 15, 2025

View reviewed changes

lslusarczyk mentioned this pull request Sep 15, 2025

memory resident limit to enabled peers intel/llvm#19257

Draft

ldorau reviewed Sep 15, 2025

View reviewed changes

pbalcer reviewed Sep 16, 2025

View reviewed changes

lslusarczyk added 10 commits September 19, 2025 13:56

Add resident device change call

e5d8739

added missing doxygen documentation

8c54695

fix documentation spelling

064d1ac

applied Rafal's comments

c4a7499

fixed compilation error

588d1af

applied second round of Rafal's comments

b24574c

applied Lukasz's comments

5983cfa

one more missed Lukasz comment applied

5cbbad2

added some tests for resident device feature

c27407a

add unittests and refactor to be backward compatible

53ec753

lslusarczyk force-pushed the memresident branch from 88f672b to 53ec753 Compare September 19, 2025 12:00

lslusarczyk added 3 commits September 19, 2025 14:07

moved umfMemoryProviderResidentDeviceChange down

01d74ba

added empty line for Rafal :)

0bc9a29

cmake format fixed

d13a3ef

bratpiorka reviewed Sep 19, 2025

View reviewed changes

lplewa requested changes Sep 19, 2025

View reviewed changes

		hParams->resident_device_handles = hDevices;
		hParams->resident_device_count = deviceCount;

Add resident device change call #1517

Are you sure you want to change the base?

Add resident device change call #1517

Uh oh!

Conversation

lslusarczyk commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lslusarczyk Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lslusarczyk Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

lslusarczyk commented Sep 11, 2025 •

edited

Loading

lslusarczyk Sep 12, 2025 •

edited

Loading

lslusarczyk Sep 15, 2025 •

edited

Loading