control additional search paths for libraries via $EPREFIX/etc/ld.so.conf rather than customizing glibc?

[boegel]

We now add /opt/eessi/lib as (primary) search path for libraries, by baking it into glibc (see EESSI/gentoo-overlay#8).

@amadio pointed out to us that a better alternative may be simply adding these paths to $EPREFIX/etc/ld.so.conf instead.

@bartoldeman: Is there a particular reason you didn't take that approach for the ComputeCanada software stack?

[bartoldeman]

Yes: ld.so.conf(.d) at runtime merely acts as input to ldconfig, the real file that matters is the cache file $EPREFIX/etc/ld.so.cache. We cannot guarantee that the contents of the additional search path are the same on the clients as on the build node.

Unless you create a local cache via a symlink but then you need to run ldconfig on the clients!

[amadio]

Yes, if the libraries differ in each client, then $EPREFIX/etc/ld.so.cache will not be up to date. I guess I understand better now why you did this in glibc, but this might be a case in which just using LD_LIBRARY_PATH on the client side would be ok.

[boegel]

@amadio All our binaries are linked with RPATH, so $LD_LIBRARY_PATH won't work at all.

[bartoldeman]

No LD_LIBRARY_PATH will do the job as well, for libraries that are not found in the RPATH.
We wanted the priority of /usr/lib64/nvidia as low as possible though but IIRC that was because of some libGL conflicts that should be behind us with libglvnd.

That leaves LD_LIBRARY_PATH as an ok choice except in some fortunately non-so-common environments that destroy it (Bazel comes to mind).

[boegel]

Our intention with the /opt/eessi/lib path(s) is to allow overriding libraries we include in our software stack though, not as fallback paths when stuff is not found.

[amadio]

There are two versions of RPATH, the old style (DT_RPATH) and new style (DT_RUNPATH), which should be default these days. The new style lets you override with LD_LIBRARY_PATH. The man page ld.so(8) has more information on this.

[boegel]

@amadio Yes, we're aware of those differences. For now, we're using DT_RPATH, partially because we want to avoid people easily shooting themselves in the foot via $LD_LIBRARY_PATH (which is sometimes set by random software packages too, creating all kinds of weird problems).

[amadio]

Ah, I see. I guess your approach to add the directory directly in glibc is indeed probably the best then.