[FEA]: Support for finding libmathdx through pathfinder

[ZzEeKkAa]

Is this a duplicate?

• I confirmed there appear to be no duplicate issues for this request and that I agree to the Code of Conduct

Area

cuda.pathfinder

Is your feature request related to a problem? Please describe.

There are multiple ways to install libmathdx (compiled mathdx) so it can be used in numba and warp:

• conda package: https://anaconda.org/conda-forge/libmathdx
• pypi: https://pypi.org/project/nvidia-libmathdx-cu12
• pypi: https://pypi.org/project/nvidia-libmathdx-cu13
• tar.gz: https://developer.nvidia.com/cublasdx-downloads

Describe the solution you'd like

cuda.pathfinder.load_nvidia_dynamic_lib("mathdx")

Describe alternatives you've considered

No response

Additional context

No response

[rwgk]

I tried this using PR #834:

conda create -y -n libmathdx python=3.12
conda activate libmathdx
conda install conda-forge::libmathdx
cd cuda_pathfinder
pip install .
python
Python 3.12.11 | packaged by conda-forge | (main, Jun  4 2025, 14:45:31) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cuda.pathfinder import load_nvidia_dynamic_lib
>>> load_nvidia_dynamic_lib("mathdx")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/rgrossekunst/forked/cuda-python/cuda_pathfinder/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 71, in load_nvidia_dynamic_lib
    return _load_lib_no_cache(libname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rgrossekunst/forked/cuda-python/cuda_pathfinder/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 44, in _load_lib_no_cache
    found.raise_if_abs_path_is_None()
  File "/home/rgrossekunst/forked/cuda-python/cuda_pathfinder/cuda/pathfinder/_dynamic_libs/find_nvidia_dynamic_lib.py", line 194, in raise_if_abs_path_is_None
    raise DynamicLibNotFoundError(f'Failure finding "{self.lib_searched_for}": {err}\n{att}')
cuda.pathfinder._dynamic_libs.load_dl_common.DynamicLibNotFoundError: Failure finding "libmathdx.so": No such file: libmathdx.so*

>>>

I.e. unfortunately PR #834 but itself does not work because conda does not create a libmathdx.so softlink:

a1u1g-mil-0536.nvidia.com:~/miniforge3/envs/libmathdx $ find . -name \*.so\* | grep mathdx
./lib/libmathdx.so.0.2.2
./lib/libmathdx.so.0

But after manually creating the softlink it works:

a1u1g-mil-0536.nvidia.com:~/miniforge3/envs/libmathdx/lib $ ln -s libmathdx.so.0 libmathdx.so

(libmathdx) a1u1g-mil-0536.nvidia.com:~/forked/cuda-python/cuda_pathfinder $ python
Python 3.12.11 | packaged by conda-forge | (main, Jun  4 2025, 14:45:31) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cuda.pathfinder import load_nvidia_dynamic_lib
>>> load_nvidia_dynamic_lib("mathdx")
LoadedDL(abs_path='/home/rgrossekunst/miniforge3/envs/libmathdx/lib/python3.12/lib-dynload/../../libmathdx.so', was_already_loaded_from_elsewhere=False, _handle_uint=95200120396080)
>>>

[rwgk]

Idea: We could

• Detect that we're running with a conda python interpreter
• Then use this simple trick to get the absolute path to the .so:

Look for the equivalent of $ORIGIN/../lib (RPATH of the conda python command), but starting with sys.executable:

(libmathdx) a1u1g-mil-0536.nvidia.com:~/forked/cuda-python/cuda_pathfinder $ python
Python 3.12.11 | packaged by conda-forge | (main, Jun  4 2025, 14:45:31) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import glob, os, sys
>>> glob.glob(os.path.join(os.path.join(os.path.dirname(os.path.dirname(sys.executable)), "lib"), "libmathdx.so*"))
['/home/rgrossekunst/miniforge3/envs/libmathdx/lib/libmathdx.so.0.2.2', '/home/rgrossekunst/miniforge3/envs/libmathdx/lib/libmathdx.so.0']

That wouldn't be very different from how we find .so files under site-packages.

[ZzEeKkAa]

does not work because conda does not create a libmathdx.so softlink

I'm pretty sure it is intended behavior for all conda-forge libraries. Don't we have to link towards <libmaname>.so.<major> by default?

https://github.com/conda-forge/cfep/pull/61/files#diff-9945e90feca9cf4f6cbccd6226992ff78ee4fc69f326455b933665813c46c7caR108-R119

https://github.com/conda-forge/libmathdx-feedstock/blob/main/recipe/meta.yaml#L59C47-L60

[rwgk]

I'm pretty sure it is intended behavior for all conda-forge libraries. Don't we have to link towards <libmaname>.so.<major> by default?

Only if there is no softlink. /usr/local/cuda has them (I just checked systematically, 13.0.0). At the moment I believe conda is the only situation we care about that needs the "simple trick" proposed above.

For site-packages, pathfinder is similarly searching for .so* files:

• cuda-python/cuda_pathfinder/cuda/pathfinder/_dynamic_libs/find_nvidia_dynamic_lib.py

  Lines 162 to 164 in b87c787

  	self.lib_searched_for = f"lib{libname}.so"
  	if self.abs_path is None:
  	self.abs_path = _find_so_using_nvidia_lib_dirs(

• cuda-python/cuda_pathfinder/cuda/pathfinder/_dynamic_libs/find_nvidia_dynamic_lib.py

  Line 31 in b87c787

  file_wild = so_basename + "*"

https://github.com/conda-forge/cfep/pull/61/files#diff-9945e90feca9cf4f6cbccd6226992ff78ee4fc69f326455b933665813c46c7caR108-R119

https://github.com/conda-forge/libmathdx-feedstock/blob/main/recipe/meta.yaml#L59C47-L60

That looks pretty standard:

{% set soversion = "14.0.1" %}
{% set soname = soversion.split('.')[0] %}
...
    - lib/libavif.so.{{ soname }}     # [linux]
    - lib/libavif.so.{{ soversion }}  # [linux]
    - Library/bin/avif-{{ soname }}.dll  # [win]

It's just that conda doesn't take the "next step down" to also provide lib/libavif.so. It surprised me a couple weeks ago. The fix at that time was this. Now I'm thinking we should handle this more generally with the "simple trick"; and maybe even undo that fix, but I'm not sure, maybe it'll be better to have both?

[kkraus14]

For the load_nvidia_dynamic_lib API we should really be trying to load based on the SONAME of the libraries that we're loading. I.E. if you compile something and link against a library, it will add a DT_NEEDED entry using the SONAME and will only find the library at runtime if there's a versioned file and won't try to load an unversioned one. If / when we introduce a find_nvidia_dynamic_lib API then trying to find an unversioned file would make more sense in my opinion.

Conda-forge packages are often split into runtime vs dev variants where the runtime variants only contain the versioned files that are referred to by the SONAME and the dev variants typically have the unversioned softlink to be used at build-time. For libmathdx there is a libmathdx package which is meant to be a runtime package, and a libmathdx-dev which is meant to be a dev package.

[ZzEeKkAa]

It's just that conda doesn't take the "next step down" to also provide lib/libavif.so. It surprised me a couple weeks ago. The fix at that time was this. Now I'm thinking we should handle this more generally with the "simple trick"; and maybe even undo that fix, but I'm not sure, maybe it'll be better to have both?

It is done on purpose, so users can install multiple major versions of the library in case they need multiple software in the env that depends on the same library, but different version.

Conda's convention is inherited from debian in some way. So it is pretty common for runtime libraries. https://www.debian.org/doc/debian-policy/ch-sharedlibs.html#run-time-shared-libraries

I'm not sure for 100%, but it is python environment that is different, cause it is expected to use only python and as result only one runtime is allowed. And since we can't have multiple major ctks loaded in one process we simply can use libraries without soversion.

[vyasr]

Given that cuda_pathfinder aims to support multiple methods of CUDA installation it still needs to be aware of the case where the user updates the minor version of the system CTK and expects everything to keep working, right? IOW you should still be using the SOVERSIONed binary?