[BUG]: KeyError in load_dl_linux.py:47 EXPECTED_LIB_SYMBOLS[libname]

[ZzEeKkAa]

Is this a duplicate?

• I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct

Type of Bug

Runtime Error

Component

cuda.pathfinder

Describe the bug

Under certain circumstances code responsible for loading ctk libraries triggers key error for a library that is not part of ctk

How to Reproduce

python -m venv .venv
source .venv/bin/activate
pip install cuda-pathfinder
export CUDA_HOME=/usr/local/cuda12.9
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
wget -q https://developer.download.nvidia.com/compute/cudss/redist/libcudss/linux-x86_64/libcudss-linux-x86_64-0.5.0.16_cuda12-archive.tar.xz
tar -xvf libcudss-linux-x86_64-0.5.0.16_cuda12-archive.tar.xz
export LD_LIBRARY_PATH=$(pwd)/libcudss-linux-x86_64-0.5.0.16_cuda12-archive/lib:$LD_LIBRARY_PATH
python -c "from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('cudss'))"

Error:

python -c "from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('cudss'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/scratch.yhavrylko_ent/Projects/nvidia/nvmath-python/.venv-sysctk12/lib/python3.11/site-packages/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 71, in load_nvidia_dynamic_lib
    return _load_lib_no_cache(libname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scratch.yhavrylko_ent/Projects/nvidia/nvmath-python/.venv-sysctk12/lib/python3.11/site-packages/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 40, in _load_lib_no_cache
    loaded = load_with_system_search(libname)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scratch.yhavrylko_ent/Projects/nvidia/nvmath-python/.venv-sysctk12/lib/python3.11/site-packages/cuda/pathfinder/_dynamic_libs/load_dl_linux.py", line 104, in load_with_system_search
    abs_path = abs_path_for_dynamic_library(libname, handle)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scratch.yhavrylko_ent/Projects/nvidia/nvmath-python/.venv-sysctk12/lib/python3.11/site-packages/cuda/pathfinder/_dynamic_libs/load_dl_linux.py", line 47, in abs_path_for_dynamic_library
    for symbol_name in EXPECTED_LIB_SYMBOLS[libname]:
                       ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
KeyError: 'cudss'

Expected behavior

Find library with respect to LD_LIBRARY_PATH or at least return proper runtime error that library was not found (not preferable, since with nvidia-cudss-cu12 it loads library properly).

Operating System

No response

nvidia-smi output

No response

[leofang]

since with nvidia-cudss-cu12 it loads library properly).

What does this mean?

[rwgk]

Find library with respect to LD_LIBRARY_PATH or at least return proper runtime error that library was not found (not preferable, since with nvidia-cudss-cu12 it loads library properly).

The library was actually found and loaded.

To figure out the absolute path under Linux, we need a trick (I researched this a lot):

cuda-python/cuda_pathfinder/cuda/pathfinder/_dynamic_libs/load_dl_linux.py

Lines 54 to 58 in b87c787

	addr = ctypes.cast(symbol, ctypes.c_void_p)
	info = DlInfo()
	if LIBDL.dladdr(addr, ctypes.byref(info)) == 0:
	raise OSError(f"dladdr failed for {libname=!r}")
	return info.dli_fname.decode() # type: ignore[no-any-return]

We need to get our hands on any one symbol in the library, and use that as a wedge (so to speak) to retrieve info.dli_fname.

This is really quite silly, but I haven't found a better way.

So, to support cudss, really all we need is one symbol name.

I'll give it a quick try.

[rwgk]

since with nvidia-cudss-cu12 it loads library properly).

What does this mean?

The abs_path_for_dynamic_library() function is called only from check_if_already_loaded_from_elsewhere() and load_with_system_search(), but not if we figured out the absolute path already by searching site-packages.

Quick update: ChatGPT 5 suggests we don't need EXPECTED_LIB_SYMBOLS. If (!) that's true, it's a game changer, so I'll try it out now.

[rwgk]

PR #834 fixes this issue (in a very general way):

Local testing:

#!/bin/bash
set -e -x
pwd
git switch linux_abs_path_from_handle_v2
git log -n 1
git clean -fdx
python3 -m venv Cp12WslVenv
. Cp12WslVenv/bin/activate
pip install --upgrade pip
cd cuda_pathfinder/
pip install .
wget -q https://developer.download.nvidia.com/compute/cudss/redist/libcudss/linux-x86_64/libcudss-linux-x86_64-0.5.0.16_cuda12-archive.tar.xz
tar -xvf libcudss-linux-x86_64-0.5.0.16_cuda12-archive.tar.xz
export LD_LIBRARY_PATH=$(pwd)/libcudss-linux-x86_64-0.5.0.16_cuda12-archive/lib:$LD_LIBRARY_PATH
python -c "from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('cudss'))"

Output:

+ pwd
/home/rgrossekunst/forked/cuda-python
+ git switch linux_abs_path_from_handle_v2
Already on 'linux_abs_path_from_handle_v2'
Your branch is up to date with 'origin/linux_abs_path_from_handle_v2'.
+ git log -n 1
commit e716f1cb2f2a5039a7d46d341aa09b9072ac8ff7
Author: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
Date:   Tue Aug 12 22:54:07 2025 -0700

    load_dl_windows.py: avoid unnecessary function-level imports (this was just an oversight)
+ git clean -fdx
Removing Cp12WslVenv/
Removing cuda_pathfinder/cuda/pathfinder/__pycache__/
Removing cuda_pathfinder/cuda/pathfinder/_dynamic_libs/__pycache__/
Removing cuda_pathfinder/cuda/pathfinder/_utils/__pycache__/
Removing cuda_pathfinder/cuda_pathfinder.egg-info/
+ python3 -m venv Cp12WslVenv
+ . Cp12WslVenv/bin/activate
++ deactivate nondestructive
++ '[' -n '' ']'
++ '[' -n '' ']'
++ hash -r
++ '[' -n '' ']'
++ unset VIRTUAL_ENV
++ unset VIRTUAL_ENV_PROMPT
++ '[' '!' nondestructive = nondestructive ']'
++ '[' linux-gnu = cygwin ']'
++ '[' linux-gnu = msys ']'
++ export VIRTUAL_ENV=/home/rgrossekunst/forked/cuda-python/Cp12WslVenv
++ VIRTUAL_ENV=/home/rgrossekunst/forked/cuda-python/Cp12WslVenv
++ _OLD_VIRTUAL_PATH=/home/rgrossekunst/forked/cuda-python/Cp12WslVenv/bin:/home/rgrossekunst/rwgk_config_nvidia/bin:.:/home/rgrossekunst/rwgk_config/bin:/home/rgrossekunst/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/snap/bin
++ PATH=/home/rgrossekunst/forked/cuda-python/Cp12WslVenv/bin:/home/rgrossekunst/forked/cuda-python/Cp12WslVenv/bin:/home/rgrossekunst/rwgk_config_nvidia/bin:.:/home/rgrossekunst/rwgk_config/bin:/home/rgrossekunst/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/snap/bin
++ export PATH
++ '[' -n '' ']'
++ '[' -z '' ']'
++ _OLD_VIRTUAL_PS1=
++ PS1='(Cp12WslVenv) '
++ export PS1
++ VIRTUAL_ENV_PROMPT='(Cp12WslVenv) '
++ export VIRTUAL_ENV_PROMPT
++ hash -r
+ pip install --upgrade pip
Requirement already satisfied: pip in ./Cp12WslVenv/lib/python3.12/site-packages (24.0)
Collecting pip
  Using cached pip-25.2-py3-none-any.whl.metadata (4.7 kB)
Using cached pip-25.2-py3-none-any.whl (1.8 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 24.0
    Uninstalling pip-24.0:
      Successfully uninstalled pip-24.0
Successfully installed pip-25.2
+ cd cuda_pathfinder/
+ pip install .
Processing /home/rgrossekunst/forked/cuda-python/cuda_pathfinder
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: cuda-pathfinder
  Building wheel for cuda-pathfinder (pyproject.toml): started
  Building wheel for cuda-pathfinder (pyproject.toml): finished with status 'done'
  Created wheel for cuda-pathfinder: filename=cuda_pathfinder-1.1.1a0-py3-none-any.whl size=17297 sha256=307a7fa1ec404d204e363864a75768a174a028caaac590a73177008c9e766a40
  Stored in directory: /tmp/pip-ephem-wheel-cache-lekk4y1h/wheels/d1/04/8c/c856bb3f85f3511d35e8d69acc9ca63f99f4d52d879aba968a
Successfully built cuda-pathfinder
Installing collected packages: cuda-pathfinder
Successfully installed cuda-pathfinder-1.1.1a0
+ wget -q https://developer.download.nvidia.com/compute/cudss/redist/libcudss/linux-x86_64/libcudss-linux-x86_64-0.5.0.16_cuda12-archive.tar.xz
+ tar -xvf libcudss-linux-x86_64-0.5.0.16_cuda12-archive.tar.xz
libcudss-linux-x86_64-0.5.0.16_cuda12-archive/
... <snip> ...
libcudss-linux-x86_64-0.5.0.16_cuda12-archive/LICENSE
++ pwd
+ export LD_LIBRARY_PATH=/home/rgrossekunst/forked/cuda-python/cuda_pathfinder/libcudss-linux-x86_64-0.5.0.16_cuda12-archive/lib:/home/rgrossekunst/z/libcudss-linux-x86_64-0.5.0.16_cuda12-archive/lib:
+ LD_LIBRARY_PATH=/home/rgrossekunst/forked/cuda-python/cuda_pathfinder/libcudss-linux-x86_64-0.5.0.16_cuda12-archive/lib:/home/rgrossekunst/z/libcudss-linux-x86_64-0.5.0.16_cuda12-archive/lib:
+ python -c 'from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('\''cudss'\''))'
LoadedDL(abs_path='/home/rgrossekunst/forked/cuda-python/cuda_pathfinder/libcudss-linux-x86_64-0.5.0.16_cuda12-archive/lib/libcudss.so', was_already_loaded_from_elsewhere=False, _handle_uint=21876640)

[ZzEeKkAa]

since with nvidia-cudss-cu12 it loads library properly).

What does this mean?

If I install cudss as a wheel - library is being loaded through the pathfinder correctly. If I install cudss as a tarball and set LD_LIBRARY_PATH, it runs into exception.