Regression in supported interconnects

[ocaisa]

I was looking at the UCX configuration in 2020.12 and I noticed that it looks like we have a regression. From EESSI/compatibility-layer#49 (comment) it looks like we should have a UCX configuration like

configure: =========================================================
configure: UCX build configuration:
configure:       Build prefix:   /home/bob/ucx/inst
configure: Preprocessor flags:   -DCPU_FLAGS="|avx" -I${abs_top_srcdir}/src -I${abs_top_builddir} -I${abs_top_builddir}/src
configure:         C compiler:   x86_64-pc-linux-gnu-gcc -O3 -g -Wall -Werror -mavx
configure:       C++ compiler:   x86_64-pc-linux-gnu-g++ -O3 -g -Wall -Werror -mavx
configure:       Multi-thread:   enabled
configure:          MPI tests:   disabled
configure:      Devel headers:   no
configure:           Bindings:   < >
configure:        UCT modules:   < ib rdmacm cma >
configure:       CUDA modules:   < >
configure:       ROCM modules:   < >
configure:         IB modules:   < >
configure:        UCM modules:   < >
configure:       Perf modules:   < >
configure: =========================================================

but in the build log for UCX (on Zen2) I see

configure: =========================================================
configure: UCX build configuration:
configure:       Build prefix:   /cvmfs/pilot.eessi-hpc.org/2020.12/software/x86_64/amd/zen2/software/UCX/1.8.0-GCCcore-9.3.0
configure: Preprocessor flags:   -DCPU_FLAGS="|avx" -I${abs_top_srcdir}/src -I${abs_top_builddir} -I${abs_top_builddir}/src
configure:         C compiler:   gcc -O3 -g -Wall -Werror -mavx
configure:       C++ compiler:   g++ -O3 -g -Wall -Werror -mavx
configure:       Multi-thread:   enabled
configure:          MPI tests:   disabled
configure:      Devel headers:   no
configure:           Bindings:   < >
configure:        UCT modules:   < ib cma >
configure:       CUDA modules:   < >
configure:       ROCM modules:   < >
configure:         IB modules:   < >
configure:        UCM modules:   < >
configure:       Perf modules:   < >
configure: =========================================================

(note the missing rdmacm)

We should probably explicitly insert what we expect from the final build (--with-rdmacm) so that configure will fail rather than build regardless. UCX in particular is critical to the stack so could do with additonal checks.

[ocaisa]

To get it get it to use rdma inside the prefix layer I needed to explicitly provide the path:

configopts = '--enable-optimizations --enable-cma --enable-mt --with-verbs --with-rdmacm=/cvmfs/pilot.eessi-hpc.org/2020.12/compat/linux/x86_64/usr --with-sysroot=/cvmfs/pilot.eessi-hpc.org/2020.12/compat/linux/x86_64'

Just setting --with-sysroot is not enough. This is probably why on some archs you might get the support and on others not, it depends on what is on the host.

[ocaisa]

This is not really an issue with the prefix layer, I'm going to move it

[ocaisa]

I also checked libfabric and I see in the configure there

checking for sysroot... no

which I would also have suspicions about.

[bedroge]

I checked our 2020.10 installation, and it has the same issue / configuration output. The output in the comment at EESSI/compatibility-layer#49 (comment) is from a manual UCX installation where I indeed explicitly passed --with-rdmacm to the configure, so we should somehow pass this to our UCX installation as well (using a hook?).

[bedroge]

I see that the configure of both libfabric and UCX allow a --with-sysroot flag:

  --with-sysroot=DIR Search for dependent libraries within DIR
                        (or the compiler's sysroot if not specified).

As the compiler has been configured with --with-sysroot set to the prefix, I assume we don't necessarily have to use this flag for these packages.

[bedroge]

This has been fixed in 2021.03:

configure:        UCT modules:   < ib rdmacm cma >

We should still have some (ReFrame?) test for this to make sure that UCX is always correctly configured in future versions, though, so let's leave this issue open to not forget about this.