[pyverbs] Don't link Python's .so at pyverbs module link stage

[heiner]

The recommended way is to not link Python's .so during module linking. The symbols will be present at runtime because the dlopen happens from Python.

Cf. pypa/cibuildwheel#639 (comment) on why FIND_PACKAGE(Python 3 ... Development) doesn't work in the manylinux images.

This is one step towards hosting a compiled version of pyverbs on PyPI (the Python Package Index). (Right now, I maintain a source distribution, which recompiles pyverbs and all of ibverbs on pip install pyverbs.)

[jgunthorpe]

I don't know about recommended, IIRC not linking is against Debian policy. There are many linker features like symbol versions that simply do not work without linking it..

Why does your environment have a problem with this? Does it not use libpython?

[heiner]

Thanks for the quick turnaround time!

The context is my attempt to build pyverbs in the manylinux image. Python modules built there are meant to be broadly compatible with most Linux distributions out there. A Python module built in a manylinux image could be uploaded to PyPI and installed via simply downloading a binary (instead of recompiling during installation).

One attempt to build pyverbs in this environment is in this branch: https://github.com/heiner/pyverbs/tree/heiner/cibuildwheel (relevant files are https://github.com/heiner/pyverbs/blob/heiner/cibuildwheel/pyproject.toml and https://github.com/heiner/pyverbs/blob/heiner/cibuildwheel/.github/workflows/publish.yml)

The first issue this runs into is that in the manylinux_2_28 image FIND_PACKAGE(Python 3 ... Development) does not work since Development means both Development.Module and Development.Embed and the latter isn't available there; as per the comment linked above Development.Embed is intentionally deleted to save space and stop a common mistake. The PR here fixes that. (Cf. also this SO answer and the second warning in the pybind11 docs here.)

I'm unsure about the Debian policy, do you happen to have a reference for that? If that's preferable, pyverbs could enable linking to the Python .so based on a cmake flag?

Alternatively, I could abandon the attempt to make pip install pyverbs work with pre-compiled binaries. The change in this PR is not sufficient for that at any rate, additional changes are required for pyverbs to link against a system-installed version of ibverbs (or shipping all ibverbs .so files together with the precompiled wheel).

[heiner]

This recommendation for the Python ecosystem is made in PEP 513:

Note that libpythonX.Y.so.1 is not on the list of libraries that a manylinux1 extension is allowed to link to.

[jgunthorpe]

If this isn't enough to achieve your goal I would prefer not to change it. I know there are distro sensitives to this topic. Come with a series that does everything you need and makes this optional only for manylinux.

[heiner]

SG!