During linking: libstdc++.so: error adding symbols: file in wrong format

[LukasvdWiel]

Dear more experienced ice sheet modeller,

I am trying to install ISSM on Matlab on our Linux cluster that runs Rocky Linux 9.4.
Rocky 9.4 has GLIBC 2.34, which is older than the GLIBC 2.35 required by the binaries of ISSM, so I am trying to compile it myself.

It configures successfully with:

./configure \
    --prefix="/trinity/opt/apps/codes/ISSM/dev10nov2025/install" \
    --with-petsc-dir="${ISSM_DIR}/externalpackages/petsc/install" \
    --includedir="${ISSM_DIR}/externalpackages/petsc/install/include" \
    --with-blas-dir="/trinity/opt/apps/libraries/Scalapack/scalapack-2.2.2" \
    --with-scalapack-dir="/trinity/opt/apps/libraries/Scalapack/scalapack-2.2.2" \
    --with-mpi-include="/trinity/opt/apps/development/dev2024/install/include" \
    --with-mpi-libflags="/trinity/opt/apps/development/dev2024/install/lib -lmpi" \
    --with-metis-dir="${PETSC_DIR}" \
    --with-parmetis-dir="${PETSC_DIR}" \
    --with-matlab-dir="/trinity/opt/apps/development/MatLab/R2025b" \
    --with-triangle-dir="${ISSM_DIR}/externalpackages/triangle/install" \
    --with-m1qn3-dir="${ISSM_DIR}/externalpackages/m1qn3/install"

but during the linking phase I keep running into:

  CXXLD    libISSMOverload.la
/usr/bin/ld: /trinity/opt/apps/development/dev2024/install/lib/../lib/libstdc++.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status

and I fail to figure out what causes this. It is typically a symptom of a conflict between binaries compiled with different compilers. I use GCC 14.1.0. Probably the Matlab binaries have been compiled with something else. Could that be the cause of this?

I have used this dev2024 environment as used in '--with-mpi-include' and '--with-mpi-libflags' for dozens of projects, and it always worked well. Could it be related to the OS already being one and half years old? Does not seem that old. Its EOL is still seven years in the future.

Also, I tried the 4.24 release version, foir comparison, even though my user preferes the development version, and 4.24 give sthe same error.

Any ideas would be greatly appreciated!

Lukas

[TidbitSoftware]

Hi Lukas,
The issue doesn't have to do with MATLAB's copy of libstdc++ as you haven't reached runtime yet. Mathieu pointed out that PETSc is compiling, so a good idea in these cases it to look up what PETSc picked up during configuration. Can you please attach $ISSM_DIR/externalpackages/petsc/src/configure.log?

[LukasvdWiel]

Of course! Thank you for taking the time to have a look.

configure.log

In the meantime I can try to compile with one of our own PETSc versions that we already have on our cluster. (update; which gave me the same libstdc++ error, so it may also be something else than PETSc)

[MathieuMorlighem]

Hi @LukasvdWiel

Looking at your log, it looks like PETSc also complained about his library

Executing: /trinity/opt/apps/development/dev2024/install/bin/mpicxx  -o /tmp/petsc-43vizdqx/config.setCompilers/conftest    /tmp/petsc-43vizdqx/config.setCompilers/conftest.o
stdout:
/usr/bin/ld: skipping incompatible /trinity/opt/apps/development/dev2024/install/lib/libstdc++.so when searching for -lstdc++
/usr/bin/ld: skipping incompatible /trinity/opt/apps/development/dev2024/install/lib/libstdc++.a when searching for -lstdc++
/usr/bin/ld: skipping incompatible /trinity/opt/apps/development/dev2024/install/lib/libgcc_s.so.1 when searching for libgcc_s.so.1
/usr/bin/ld: skipping incompatible /trinity/opt/apps/development/dev2024/install/lib/libgcc_s.so.1 when searching for libgcc_s.so.1
/usr/bin/ld: skipping incompatible /trinity/opt/apps/development/dev2024/install/lib/libgcc_s.so.1 when searching for libgcc_s.so.1
/usr/bin/ld: skipping incompatible /trinity/opt/apps/development/dev2024/install/lib/libgcc_s.so.1 when searching for libgcc_s.so.1
Possible ERROR while running linker:exit code 0

It ended up defining this macro:

 Defined make macro "PETSC_EXTERNAL_LIB_BASIC" to "-Wl,-rpath,/trinity/opt/apps/codes/ISSM/dev10nov2025/externalpackages/petsc/install/lib -L/trinity/opt/apps/codes/ISSM/dev10nov2025/externalpackages/petsc/install/lib -Wl,-rpath,/trinity/opt/apps/development/dev2024/install/lib -L/trinity/opt/apps/development/dev2024/install/lib -Wl,-rpath,/trinity/opt/apps/development/dev2024/install/lib64 -L/trinity/opt/apps/development/dev2024/install/lib64 -Wl,-rpath,/trinity/opt/apps/development/dev2024/install/lib/gcc/x86_64-redhat-linux/14.1.0 -L/trinity/opt/apps/development/dev2024/install/lib/gcc/x86_64-redhat-linux/14.1.0 -Wl,-rpath,/trinity/opt/apps/development/dev2024/packages/julia-1.11.1/lib -L/trinity/opt/apps/development/dev2024/packages/julia-1.11.1/lib -ldmumps -lmumps_common -lpord -lpthread -lscalapack -lflapack -lfblas -lparmetis -lmetis -lm -lz -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -lquadmath"

Could you tell us which one of these directories has libstdc++.so? (probably /trinity/opt/apps/development/dev2024/install/lib/gcc/x86_64-redhat-linux/14.1.0 ?)

You are also welcome to use your own PETSc module, but it looks like you have 2 gcc libraries that are incompatible with each other (maybe one is 32-bits and the other 64), and that's what is creating the problem.
Cheers
Mathieu

[LukasvdWiel]

Hi Mathieu,

thank you for investigating this and helping me out.
My libstdc++ is in: /trinity/opt/apps/development/dev2024/install/lib/libstdc++.so.6.0.33.
but also in: /trinity/opt/apps/development/dev2024/install/lib64/libstdc++.so.6.0.33.

I see there are components that link to the 32 bits library, and some that link to the 64 bits version.
I suspect that is the direction that I need to be looking for.
Thank you for pointing my nose in the right direction. :D
I wil let you know when I solved it.

Out of genuine curiosity, what OS are your running us that already has GLIBC 2.35?

Enjoy your day!
Lukas

[LukasvdWiel]

Sadly this train of thought has not led to a solution. Tried with a more Spartan environment, to make sure that there was only one single libstdc++ to link to, and did not work either. Played around with different compiler versions, with our system compiler, with a manually installed metis, or with a metis that comes with PETSc. With a manually installed PETSc, or one that can be installed using the ISSM external package installer. Nothing worked.

I am at a loss for ideas. We are at the point of running it on the national supercomputer which hopefully has more capable software-people to install it, but this is still gnawing at my confidence. (or a bleeding edge OS with a more modern GLIBC which will make everything a lot easier)

What are the configuration options that you use to compile ISSM to create the binaries you offer?
Maybe I can get a hint from that information.

[MathieuMorlighem]

Hi @LukasvdWiel

Have you tried recompiling your code with

export LDFLAGS="${LDFLAGS} -Wl,-L/trinity/opt/apps/development/dev2024/install/lib/ -lstdc++

(you can put this at the top of your configure.sh file. When you compile you can do make V=1 to make sure it is properly picked up. Let us know how it goes
Mathieu

[TidbitSoftware]

@LukasvdWiel

What are the configuration options that you use to compile ISSM to create the binaries you offer?

I don't believe this will help as we use static libraries wherever possible in our precompiled packages. That said, if you want to look at any of our build recipes, check out the jenkins/ directory in your copy of the ISSM repo. Anything prefixed with ross-debian_linux- is one of our standard Linux builds, but you might also consult the linux64_caladan ones.

[LukasvdWiel]

@MathieuMorlighem Does this suggestion of that path imply that the ISSM PETSc install should come with its own lstdc++ ?
Neither my ISSM PETSc nor our system PETSc installs have generated their own lstdc++ versions.

[MathieuMorlighem]

Sorry @LukasvdWiel somehow I put the wrong path! updated now

[LukasvdWiel]

Thanks! Sadly I does not help met to get beyond

libtool: link: /trinity/opt/apps/development/dev2024/install/bin/mpic++ -fPIC -DPIC -shared -nostdlib
/lib/../lib64/crti.o
[*** tons of ISSM of object files ***]
 -Wl,-rpath
-Wl,/trinity/opt/apps/development/dev2024/install/lib -Wl,-rpath
-Wl,/trinity/opt/apps/development/dev2024/install/lib/../lib -Wl,-rpath
-Wl,/trinity/opt/apps/development/dev2024/install/lib -Wl,-rpath
-Wl,/trinity/opt/apps/development/dev2024/install/lib/../lib
-L/trinity/opt/apps/codes/ISSM/dev21nov2025/externalpackages/petsc/install/lib -lpetsc
-L/trinity/opt/apps/development/dev2024/packages/metis-5.1.0/lib -lmetis
-L/trinity/opt/apps/codes/ISSM/dev21nov2025/externalpackages/m1qn3/install -lm1qn3 -lddot
-L/trinity/opt/apps/development/dev2024/install/lib
-L/trinity/opt/apps/development/dev2024/install/lib/../lib64
-L/trinity/opt/apps/development/dev2024/install/lib64/../lib64
-L/trinity/opt/apps/development/dev2024/install/lib/gcc/x86_64-redhat-linux/14.1.0
-L/trinity/opt/apps/development/dev2024/install/lib/gcc/x86_64-redhat-linux/14.1.0/../../../../lib64
-L/lib/../lib64 -L/usr/lib/../lib64 -L/trinity/opt/apps/development/dev2024/packages/julia-1.11.1/lib
-L/trinity/opt/apps/development/dev2024/install/lib64
-L/trinity/opt/apps/development/dev2024/install/lib/gcc/x86_64-redhat-linux/14.1.0/../../..
/trinity/opt/apps/development/dev2024/install/lib/libmpi.so -lpthread
/trinity/opt/apps/development/dev2024/install/lib/libopen-pal.so -lucp -lucs -lucm -luct
/trinity/opt/apps/development/dev2024/install/lib/libpmix.so -ldl -levent_core -levent_pthreads -lhwloc
/trinity/opt/apps/development/dev2024/install/lib/../lib/libstdc++.so -lm -lc -lgcc_s
/trinity/opt/apps/development/dev2024/install/lib/gcc/x86_64-redhat-linux/14.1.0/crtendS.o
/lib/../lib64/crtn.o -g -O2 -g -O2 -Wl,-L/trinity/opt/apps/development/dev2024/install/lib -Wl,-soname
-Wl,libISSMCore.so -o .libs/libISSMCore.so

/usr/bin/ld: /trinity/opt/apps/development/dev2024/install/lib/../lib/libstdc++.so: error adding symbols: file 
in wrong format

collect2: error: ld returned 1 exit status

It all looks consistent, but the error does not specify any details of what file is exactly causing the conflict.

[MathieuMorlighem]

can you tell us what you get when you do

cat $ISSM_DIR/externalpackages/petsc/install/lib/petsc/conf/petscvariables | grep PETSC_EXTERNAL_LIB_BASIC

PETSc managed to compile so it must have found the right libraries...