Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update easyblocks to use EasyConfig.parallel property #3557

Open
wants to merge 3 commits into
base: 5.0.x
Choose a base branch
from

Conversation

Flamefire
Copy link
Contributor

Don't use the deprecated ec['parallel'] value anymore.

Introduce a property to ConfigureMake to get the commonly used -j <n> string.

Verify each use converting them to fStrings. Partially because it makes it easier to read but also to let me verify I had checked every place where it is used.

Requires

@Flamefire
Copy link
Contributor Author

I prepared a list file of ECs changed in easybuilders/easybuild-easyconfigs#19375 that are still in 5.0x to test this with which I'm currently doing over the next 3 days (some downtime here)

Don't use the deprecated `ec['parallel']` value anymore.
Introduce a property to `ConfigureMake` to get the commonly used `-j <n>` string.
Verify each use converting them to fStrings.
@bartoldeman
Copy link
Contributor

Test report by @bartoldeman

Overview of tested easyconfigs (in order)

  • SUCCESS Boost-1.85.0-GCC-13.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node1.int.archimedes.c3.ca - Linux Rocky Linux 9.4 (Blue Onyx), x86_64, Intel Xeon Processor (Skylake, IBRS) (skylake_avx512), Python 3.11.4
See https://gist.github.com/bartoldeman/ba71bcc9ec86bb2c427da2799f91b1c3 for a full test report.

@bartoldeman
Copy link
Contributor

Test report by @bartoldeman

Overview of tested easyconfigs (in order)

  • SUCCESS libevent-2.1.12-GCCcore-13.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node1.int.archimedes.c3.ca - Linux Rocky Linux 9.4 (Blue Onyx), x86_64, Intel Xeon Processor (Skylake, IBRS) (skylake_avx512), Python 3.11.4
See https://gist.github.com/bartoldeman/9a97797d11dd316e5b42e9f9657c971d for a full test report.

@bartoldeman
Copy link
Contributor

Test report by @bartoldeman

Overview of tested easyconfigs (in order)

  • SUCCESS OpenMPI-5.0.3-GCC-13.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node1.int.archimedes.c3.ca - Linux Rocky Linux 9.4 (Blue Onyx), x86_64, Intel Xeon Processor (Skylake, IBRS) (skylake_avx512), Python 3.11.4
See https://gist.github.com/bartoldeman/72323a55504c8938a4e784c19cb12339 for a full test report.

@bartoldeman
Copy link
Contributor

bartoldeman commented Feb 12, 2025

test checkboard:

  • adf.py
  • aomp.py
  • bazel.py
  • berkeleygw.py
  • boost.py
  • cargo.py
  • clang.py
  • configuremake.py
  • cp2k.py
  • dm_reverb.py
  • flook.py
  • gamess_us.py
  • gcc.py
  • gromacs.py
  • hadoop.py
  • impi.py
  • jaxlib.py
  • mesonninja.py
  • molpro.py
  • mrtrix.py
  • namd.py
  • neuron.py
  • nwchem.py
  • openblas.py
  • openfoam.py
  • openmpi.py
  • parmetis.py
  • pdt.py
  • perl.py
  • petsc.py
  • psi.py
  • pytorch.py
  • qt.py
  • quantumespresso.py
  • rosetta.py
  • scalapack.py
  • scipy.py
  • scons.py
  • siesta.py
  • slepc.py
  • tensorflow.py
  • tensorflow_compression.py
  • wrf.py

@bartoldeman
Copy link
Contributor

Test report by @bartoldeman

Overview of tested easyconfigs (in order)

Build succeeded for 0 out of 1 (1 easyconfigs in total)
node1.int.archimedes.c3.ca - Linux Rocky Linux 9.4 (Blue Onyx), x86_64, Intel Xeon Processor (Skylake, IBRS) (skylake_avx512), Python 3.11.4
See https://gist.github.com/bartoldeman/a2c0b7bfa4fecb54c2888f358cf880b1 for a full test report.

@bartoldeman
Copy link
Contributor

@Flamefire Perl fails with

== FAILED: Installation ended unsuccessfully: Parallelism in EasyConfig not set yet. Need to call the easyblocks set_parallel
first. (took 7 mins 39 secs)

@bartoldeman
Copy link
Contributor

Test report by @bartoldeman

Overview of tested easyconfigs (in order)

  • SUCCESS OpenBLAS-0.3.27-GCC-13.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node1.int.archimedes.c3.ca - Linux Rocky Linux 9.4 (Blue Onyx), x86_64, Intel Xeon Processor (Skylake, IBRS) (skylake_avx512), Python 3.11.4
See https://gist.github.com/bartoldeman/d32b30ed94e2877a85dbd582a28282c3 for a full test report.

@bartoldeman
Copy link
Contributor

Test report by @bartoldeman

Overview of tested easyconfigs (in order)

  • SUCCESS ScaLAPACK-2.2.0-gompi-2024a-fb.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node1.int.archimedes.c3.ca - Linux Rocky Linux 9.4 (Blue Onyx), x86_64, Intel Xeon Processor (Skylake, IBRS) (skylake_avx512), Python 3.11.4
See https://gist.github.com/bartoldeman/e2f1ddecf8ea9d37bf3beff992451d8a for a full test report.

@Flamefire
Copy link
Contributor Author

GCCcore failures: Sanity check failed: No '(RPATH)' found in 'readelf -d'

Doesn't seem to be caused by this. Can you try without this PR in the same environment?

@Flamefire
Copy link
Contributor Author

@Flamefire Perl fails with

== FAILED: Installation ended unsuccessfully: Parallelism in EasyConfig not set yet. Need to call the easyblocks set_parallel
first. (took 7 mins 39 secs)

I fear there is some larger underlying issue: Extensions can set maxparallel too in their options. It is possible that we don't handle that right now. Have to verify that. Not sure what we'd use. Probably just what the parent EC has determined.

@Flamefire
Copy link
Contributor Author

I added a new commit to the framework PR that should fix the Perl build

@bartoldeman
Copy link
Contributor

Test report by @bartoldeman

Overview of tested easyconfigs (in order)

  • SUCCESS GCCcore-14.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node1.int.archimedes.c3.ca - Linux Rocky Linux 9.4 (Blue Onyx), x86_64, Intel Xeon Processor (Skylake, IBRS) (skylake_avx512), Python 3.11.4
See https://gist.github.com/bartoldeman/94d6d66f757b2f4703da79b472e01266 for a full test report.

@bartoldeman
Copy link
Contributor

GCC fail was a local issue due to the special RPATH processing we do, I just rebuilt with --disable-rpath and it went ok.

@bartoldeman
Copy link
Contributor

Test report by @bartoldeman

Overview of tested easyconfigs (in order)

  • SUCCESS Perl-5.38.2-GCCcore-13.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node1.int.archimedes.c3.ca - Linux Rocky Linux 9.4 (Blue Onyx), x86_64, Intel Xeon Processor (Skylake, IBRS) (skylake_avx512), Python 3.11.4
See https://gist.github.com/bartoldeman/19efe7cf58d8c177a12557f8679e5e3d for a full test report.

@bartoldeman
Copy link
Contributor

I added a new commit to the framework PR that should fix the Perl build

Thanks, it works now!

@bartoldeman
Copy link
Contributor

Test report by @bartoldeman

Overview of tested easyconfigs (in order)

  • SUCCESS ATK-2.38.0-GCCcore-13.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node1.int.archimedes.c3.ca - Linux Rocky Linux 9.4 (Blue Onyx), x86_64, Intel Xeon Processor (Skylake, IBRS) (skylake_avx512), Python 3.11.4
See https://gist.github.com/bartoldeman/93cd5df9acec5f97e55029d5a876f760 for a full test report.

@Flamefire
Copy link
Contributor Author

There was another fix required for ECs using extensions with --sanity-check-only. Updated the frameworks PR.

@bartoldeman
Copy link
Contributor

Test report by @bartoldeman

Overview of tested easyconfigs (in order)

Build succeeded for 0 out of 1 (1 easyconfigs in total)
node1.int.archimedes.c3.ca - Linux Rocky Linux 9.4 (Blue Onyx), x86_64, Intel Xeon Processor (Skylake, IBRS) (skylake_avx512), Python 3.11.4
See https://gist.github.com/bartoldeman/15d23a0dc255f279c8bb52fb1a087bcf for a full test report.

@Flamefire
Copy link
Contributor Author

@bartoldeman

src/main/tools/linux-sandbox-pid1.cc:548: "execvp(/tmp/eb-c98ftzb8/tmpyrheqmcc/rpath_wrappers/gcc_wrapper/gcc, 0x20c53b0)": No such file or directory

@bartoldeman
Copy link
Contributor

Test report by @bartoldeman

Overview of tested easyconfigs (in order)

  • SUCCESS Bazel-7.4.1-GCCcore-13.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node1.int.archimedes.c3.ca - Linux Rocky Linux 9.4 (Blue Onyx), x86_64, Intel Xeon Processor (Skylake, IBRS) (skylake_avx512), Python 3.11.4
See https://gist.github.com/bartoldeman/6a60a1143a984685b6f2cb0b695dc5e5 for a full test report.

@bartoldeman
Copy link
Contributor

@bartoldeman

src/main/tools/linux-sandbox-pid1.cc:548: "execvp(/tmp/eb-c98ftzb8/tmpyrheqmcc/rpath_wrappers/gcc_wrapper/gcc, 0x20c53b0)": No such file or directory

yes Bazel doesn't like it when I have the rpath wrappers or ccache enabled. It may be something specific to our environment, not related to your PR

@bartoldeman
Copy link
Contributor

Test report by @bartoldeman

Overview of tested easyconfigs (in order)

  • SUCCESS bcrypt-4.0.1-GCCcore-12.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node1.int.archimedes.c3.ca - Linux Rocky Linux 9.4 (Blue Onyx), x86_64, Intel Xeon Processor (Skylake, IBRS) (skylake_avx512), Python 3.11.4
See https://gist.github.com/bartoldeman/26d182dd346d4df6583c7faf8eceab75 for a full test report.

@Flamefire
Copy link
Contributor Author

Test report by @Flamefire

Overview of tested easyconfigs (in order)

  • SUCCESS dftd3-lib-0.9-intel-compilers-2021.2.0.eb
  • SUCCESS flook-0.8.1-intel-compilers-2021.4.0.eb
  • SUCCESS kim-api-2.3.0-intel-compilers-2023.1.0.eb
  • SUCCESS libmbd-0.10.4-intel-2021a.eb
  • SUCCESS libxc-5.2.3-intel-compilers-2022.1.0.eb
  • FAIL (build issue) QuantumESPRESSO-7.3-intel-2023a.eb (partial log available at https://gist.github.com/Flamefire/b0e80049a5b456f605e105114521670e)
  • SUCCESS SuiteSparse-5.10.1-intel-2021a-METIS-5.1.0.eb
  • SUCCESS Yambo-5.0.4-intel-2021a.eb

Build succeeded for 7 out of 8 (8 easyconfigs in total)
i7153 - Linux Rocky Linux 8.9 (Green Obsidian), x86_64, AMD EPYC 7702 64-Core Processor (zen2), Python 3.8.17
See https://gist.github.com/Flamefire/506ba13bd63dde20d7ada8d7e3e52d73 for a full test report.

@bartoldeman
Copy link
Contributor

Test report by @bartoldeman

Overview of tested easyconfigs (in order)

  • SUCCESS juliaup-1.17.9-GCCcore-12.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node1.int.archimedes.c3.ca - Linux Rocky Linux 9.4 (Blue Onyx), x86_64, Intel Xeon Processor (Skylake, IBRS) (skylake_avx512), Python 3.11.4
See https://gist.github.com/bartoldeman/5dc0e901ec2301e9e2401d0779fb6b2a for a full test report.

@Flamefire
Copy link
Contributor Author

Test report by @Flamefire

Overview of tested easyconfigs (in order)

  • SUCCESS AMOS-3.1.0-foss-2021b.eb
  • SUCCESS AMOS-3.1.0-foss-2023a.eb
  • SUCCESS APR-util-1.6.1-GCCcore-10.3.0.eb
  • SUCCESS APR-util-1.6.3-GCCcore-13.3.0.eb
  • SUCCESS AutoDock-GPU-1.5.3-GCC-10.3.0-CUDA-11.3.1.eb
  • SUCCESS AutoDock-GPU-1.5.3-GCC-11.3.0-CUDA-11.7.0.eb
  • SUCCESS bioawk-1.0-GCC-10.3.0.eb
  • SUCCESS bioawk-1.0-GCC-11.2.0.eb
  • SUCCESS Bracken-2.6.2-GCCcore-11.2.0.eb
  • SUCCESS Bracken-2.9-GCCcore-10.3.0.eb
  • SUCCESS ctffind-4.1.14-foss-2021a-CUDA-11.3.1.eb
  • SUCCESS ctffind-4.1.14-foss-2023a.eb
  • SUCCESS Delft3D-4.04.01-foss-2022a-FLOW.eb
  • FAIL (build issue) Delft3D-4.04.01-foss-2022a-FLOW.eb (partial log available at https://gist.github.com/Flamefire/b51c8bd1779c406163b677f3b67eb743)
  • SUCCESS Elk-7.2.42-foss-2021a.eb
  • SUCCESS Elk-8.5.2-foss-2022a.eb
  • SUCCESS EMBOSS-6.6.0-foss-2021b.eb
  • SUCCESS Exonerate-2.4.0-GCC-10.3.0.eb
  • SUCCESS Exonerate-2.4.0-GCC-12.3.0.eb
  • SUCCESS fastml-2.3-GCC-11.3.0.eb
  • SUCCESS FDMNES-2024-02-29-gomkl-2023a.eb
  • SUCCESS fermi-lite-20190320-GCCcore-12.3.0.eb
  • SUCCESS Ferret-7.6.0-foss-2022a.eb
  • SUCCESS fftlib-20170628-gompi-2022a.eb
  • SUCCESS Filtlong-0.2.1-GCC-10.3.0.eb
  • SUCCESS Filtlong-0.2.1-GCC-12.3.0.eb
  • SUCCESS flook-0.8.1-GCC-10.3.0.eb
  • SUCCESS g2clib-1.6.3-GCCcore-10.3.0.eb
  • SUCCESS g2lib-3.2.0-GCCcore-13.2.0.eb
  • SUCCESS GDMA-2.3.3_20230603-GCC-12.3.0.eb
  • SUCCESS GLI-4.5.31-GCCcore-12.2.0.eb
  • SUCCESS how_are_we_stranded_here-1.0.1-foss-2021b.eb
  • SUCCESS HPCC-1.5.0-foss-2022a.eb
  • SUCCESS IMB-2021.3-gompi-2021b.eb
  • SUCCESS IMB-2021.3-iimpi-2022b.eb
  • SUCCESS jax-0.2.24-foss-2021a-CUDA-11.3.1.eb
  • SUCCESS jax-0.3.9-foss-2021a.eb
  • SUCCESS Jellyfish-2.3.0-GCC-10.3.0.eb
  • SUCCESS Jellyfish-2.3.0-GCC-12.2.0.eb
  • SUCCESS kallisto-0.48.0-gompi-2021a.eb
  • SUCCESS kallisto-0.50.1-gompi-2022b.eb
  • SUCCESS kim-api-2.2.1-GCC-10.3.0.eb
  • SUCCESS libharu-2.3.0-foss-2021a.eb
  • SUCCESS libharu-2.3.0-GCCcore-11.3.0.eb
  • SUCCESS libmbd-0.10.4-foss-2021a.eb
  • SUCCESS libRmath-4.1.2-GCCcore-11.2.0.eb
  • SUCCESS libRmath-4.2.1-GCCcore-11.3.0.eb
  • SUCCESS libunistring-0.9.10-GCCcore-10.3.0.eb
  • SUCCESS libunistring-1.2-GCCcore-13.3.0.eb
  • SUCCESS libxc-4.3.4-GCC-10.3.0.eb
  • SUCCESS MIGRATE-N-5.0.4-foss-2021b.eb
  • SUCCESS Molden-7.1-GCCcore-11.3.0.eb
  • SUCCESS Molden-7.3-GCCcore-12.3.0.eb
  • SUCCESS motif-2.3.8-GCCcore-10.3.0.eb
  • SUCCESS motif-2.3.8-GCCcore-12.3.0.eb
  • SUCCESS netCDF-Fortran-4.5.3-gompi-2021a.eb
  • SUCCESS netCDF-Fortran-4.6.1-iimpi-2023b.eb
  • SUCCESS NSS-3.65-GCCcore-10.3.0.eb
  • SUCCESS NSS-3.94-GCCcore-13.2.0.eb
  • SUCCESS Oases-20180312-GCC-12.3.0.eb
  • SUCCESS OCaml-4.14.0-GCC-11.3.0.eb
  • SUCCESS PortAudio-19.7.0-foss-2022a.eb
  • SUCCESS PsiCLASS-1.0.3-GCC-11.2.0.eb
  • SUCCESS PSIPRED-4.02-GCC-12.3.0.eb
  • SUCCESS Qhull-2020.2-GCCcore-10.3.0.eb
  • SUCCESS Qhull-2020.2-GCCcore-13.3.0.eb
  • SUCCESS qrupdate-1.1.2-GCCcore-11.2.0.eb
  • SUCCESS qrupdate-1.1.2-GCCcore-11.3.0.eb
  • SUCCESS QuantumESPRESSO-6.7-foss-2021a.eb
  • SUCCESS QuickPIC-20210224-gompi-2021b.eb
  • SUCCESS SBCL-2.2.1-GCCcore-10.3.0.eb
  • SUCCESS SBCL-2.4.1-GCCcore-12.3.0.eb
  • SUCCESS STAR-2.7.10a_alpha_220601-GCC-10.3.0.eb
  • SUCCESS STAR-2.7.9a-GCC-11.2.0.eb
  • FAIL (build issue) TopHat-2.1.2-GCC-11.2.0-Python-2.7.18.eb (partial log available at https://gist.github.com/Flamefire/afed383e1b3adda7b416fa18eb1bab97)
  • SUCCESS TopHat-2.1.2-GCC-11.3.0-Python-2.7.18.eb
  • SUCCESS tRNAscan-SE-2.0.12-foss-2022b.eb
  • SUCCESS UDUNITS-2.2.28-GCCcore-10.3.0.eb
  • SUCCESS UDUNITS-2.2.28-GCCcore-13.2.0.eb
  • SUCCESS waLBerla-6.1-foss-2021a.eb
  • SUCCESS waLBerla-6.1-foss-2022b.eb
  • SUCCESS EMBOSS-6.6.0-foss-2021a.eb

Build succeeded for 80 out of 82 (82 easyconfigs in total)
i7092 - Linux Rocky Linux 8.9 (Green Obsidian), x86_64, AMD EPYC 7702 64-Core Processor (zen2), Python 3.8.17
See https://gist.github.com/Flamefire/f71892f8ab720e9a965aeae0bb163dc6 for a full test report.

@Flamefire
Copy link
Contributor Author

Flamefire commented Feb 19, 2025

Pretty much all ECs changed in the parallel easyconfigs PR passed.

  • Delft3D-4.04.01-foss-2022a-FLOW.eb fails due to a bug in framework: Environment variable CXX set to mpicxx mpicxx (previously undefined). This does only happen when building multiple ECs with that one at once. Edit: Actually I added that EC twice, the 2nd fails due to a Bug in EB
  • TopHat-2.1.2-GCC-11.2.0-Python-2.7.18.eb uses static linking without specifying the OS dependencies. It is the only version of TopHat that does this, the others work fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Breaking changes
Development

Successfully merging this pull request may close these issues.

3 participants