Firedrake tests not passing

[Brazovskis]

Good afternoon,

I have been facing issues with getting a working installation of Firedrake on an Ubuntu 22.04 machine with Intel Xeon W7 machine.

The installation succeeds with no errors, however, when testing the installation with
pytest tests/regression/ -k "poisson_strong or stokes_mini or dg_advection"
I get a number of tests failing:

====================================================== short test summary info ======================================================
FAILED tests/regression/test_poisson_strong_bcs.py::test_poisson_analytic[params1-1-True] - assert 0.9562263984999115 < 1e-09
FAILED tests/regression/test_poisson_strong_bcs.py::test_poisson_analytic[params2-2-False] - assert 0.583520056710691 < 1e-09
FAILED tests/regression/test_poisson_strong_bcs.py::test_poisson_analytic[params3-2-True] - firedrake.exceptions.ConvergenceError: Nonlinear solve failed to converge after 25 nonlinear iterations.
FAILED tests/regression/test_poisson_strong_bcs.py::test_poisson_analytic_linear[params1-1-True] - assert 1.857809736663187 < 5e-06
FAILED tests/regression/test_poisson_strong_bcs.py::test_poisson_analytic_linear[params2-2-False] - assert 6.649588457517463 < 5e-06
FAILED tests/regression/test_poisson_strong_bcs.py::test_poisson_analytic_linear[params3-2-True] - assert 5.331332146018825 < 5e-06
FAILED tests/regression/test_poisson_strong_bcs_nitsche.py::test_poisson_nitsche[1-True] - assert 3.0350111740039 < 0.001
FAILED tests/regression/test_poisson_strong_bcs_nitsche.py::test_poisson_nitsche[2-False] - assert 230.1658349216025 < 0.001
FAILED tests/regression/test_poisson_strong_bcs_nitsche.py::test_poisson_nitsche[2-True] - assert 5.309348381917987 < 0.001

A similar issue has been discussed in the Firedrake Slack channel before without a definitive solution identified. Following suggestions in said thread, I also provide the following outputs.

Output of ldd $VIRTUAL_ENV/lib/python*/site-packages/mpi4py/MPI.*.so:

linux-vdso.so.1 (0x00007ffccabdd000)
libmpi.so.12 => /home/emils/firedrake/src/petsc/default/lib/libmpi.so.12 (0x00007fef6a600000)
libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x00007fef6a200000)
libhwloc.so.15 => /home/emils/firedrake/src/petsc/default/lib/libhwloc.so.15 (0x00007fef6ab5b000)
/lib64/ld-linux-x86-64.so.2 (0x00007fef6ad1b000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fef6a519000)

Output of PETSc make check:

Running PETSc check examples to verify correct installation
Using PETSC_DIR=/home/emils/firedrake/src/petsc and PETSC_ARCH=default
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes
C/C++ example src/snes/tutorials/ex19 run successfully with HYPRE
C/C++ example src/snes/tutorials/ex19 run successfully with MUMPS
3,5c3,4
<   1 SNES Function norm 3.32845e-06
<   2 SNES Function norm < 1.e-11
< Number of SNES iterations = 2
---
>   1 SNES Function norm 0.0404122
> Number of SNES iterations = 1
/home/emils/firedrake/src/petsc/src/snes/tutorials
Possible problem with ex19 running with SuiteSparse, diffs above
=========================================
2c2
< Number of SNES iterations = 2
---
> Number of SNES iterations = 0
/home/emils/firedrake/src/petsc/src/snes/tutorials
Possible problem with ex19 running with SuperLU_DIST, diffs above
=========================================
C/C++ example src/vec/vec/tests/ex47 run successfully with HDF5
Completed PETSc check examples
Error while running make check
gmake[1]: *** [makefile:131: check] Error 1
make: *** [GNUmakefile:17: check] Error 2

The Firedrake install log:
firedrake-install.log

The PETSc src/petsc/configure.log log:
configure.log

Any help would be greatly appreciated!

[connorjward]

The fact that make check is failing suggests that the issue is with PETSc. Could you try compiling PETSc separately? The steps you need to follow are:

1. Make sure that the current venv isn't activated.
2. Clone our fork of PETSc
3. Run (inside the cloned directory)

./configure --with-shared-libraries=1 --download-cmake --download-pastix --download-bison --download-netcdf --download-pnetcdf --download-mpich --download-scalapack --download-mumps --download-suitesparse --download-metis --download-hdf5 --download-ptscotch --download-chaco --download-hypre --with-zlib --download-hwloc --with-c2html=0 --with-debugging=0 --with-fortran-bindings=0 --download-superlu_dist

4. Run make
5. Run make check

If this fails to pass then the issue is definitely with PETSc and we can iterate from there.

[Brazovskis]

Getting the same issues when following the steps you provided:

Running PETSc check examples to verify correct installation
Using PETSC_DIR=/home/emils/petsc and PETSC_ARCH=arch-linux-c-opt
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes
C/C++ example src/snes/tutorials/ex19 run successfully with HYPRE
C/C++ example src/snes/tutorials/ex19 run successfully with MUMPS
3,5c3,4
<   1 SNES Function norm 3.32845e-06
<   2 SNES Function norm < 1.e-11
< Number of SNES iterations = 2
---
>   1 SNES Function norm 0.0404122
> Number of SNES iterations = 1
/home/emils/petsc/src/snes/tutorials
Possible problem with ex19 running with SuiteSparse, diffs above
=========================================
2c2
< Number of SNES iterations = 2
---
> Number of SNES iterations = 0
/home/emils/petsc/src/snes/tutorials
Possible problem with ex19 running with SuperLU_DIST, diffs above
=========================================
C/C++ example src/vec/vec/tests/ex47 run successfully with HDF5
Completed PETSc check examples
Error while running make check
gmake[1]: *** [makefile:131: check] Error 1
make: *** [GNUmakefile:17: check] Error 2

[connorjward]

OK. So the issue is definitely with PETSc. Can you now try running exactly the same steps but with the most recent PETSc (git clone https://gitlab.com/petsc/petsc.git)? If the same errors occur then an issue needs to be submitted to PETSc, if it passes then we need to update our fork (which we updated very recently).

[Brazovskis]

Yes, seemingly still getting the same errors with the most recent PETSc version:

Running PETSc check examples to verify correct installation
Using PETSC_DIR=/home/emils/petsc and PETSC_ARCH=arch-linux-c-opt
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes
C/C++ example src/snes/tutorials/ex19 run successfully with HYPRE
C/C++ example src/snes/tutorials/ex19 run successfully with MUMPS
3,5c3,4
<   1 SNES Function norm 3.32845e-06
<   2 SNES Function norm < 1.e-11
< Number of SNES iterations = 2
---
>   1 SNES Function norm 0.0404122
> Number of SNES iterations = 1
/home/emils/petsc/src/snes/tutorials
Possible problem with ex19 running with SuiteSparse, diffs above
=========================================
2c2
< Number of SNES iterations = 2
---
> Number of SNES iterations = 0
/home/emils/petsc/src/snes/tutorials
Possible problem with ex19 running with SuperLU_DIST, diffs above
=========================================
C/C++ example src/vec/vec/tests/ex47 run successfully with HDF5
Completed PETSc check examples
Error while running make check
gmake[1]: *** [makefile:131: check] Error 1
make: *** [GNUmakefile:17: check] Error 2

[connorjward]

OK. Would you be able to submit an issue to PETSc then (and share a link to it here)? Sorry we can't be of further help.

[Brazovskis]

I've submitted the issue to PETSc here. Thank you for your help!

[Brazovskis]

Adding a --download-fblaslapack flag to ./configure when installing PETSc seems to have solved the issue. Here's the make check output:

Running PETSc check examples to verify correct installation
Using PETSC_DIR=/home/emils/petsc and PETSC_ARCH=arch-linux-c-opt
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes
C/C++ example src/snes/tutorials/ex19 run successfully with HYPRE
C/C++ example src/snes/tutorials/ex19 run successfully with MUMPS
C/C++ example src/snes/tutorials/ex19 run successfully with SuiteSparse
C/C++ example src/snes/tutorials/ex19 run successfully with SuperLU_DIST
C/C++ example src/vec/vec/tests/ex47 run successfully with HDF5
Completed PETSc check examples

Adding the flag in the firedrake-install script, and then testing the installation with pytest tests/regression/ -k "poisson_strong or stokes_mini or dg_advection", all the tests seem to pass.

Adding the flag --download-fblaslapack to petsc_options in the firedrake-install file also works.

Thank you for your help!