Building on top of EESSI

[ocaisa]

I tested building on top of the EESSI setup this week and I think there is scope to add an easyblock to EasyBuild that will set up a module for you to allow you to easily do this. If you look at the current build script we use, there are only a few environment variables that are required:

export EASYBUILD_IGNORE_OSDEPS=1
export EASYBUILD_SYSROOT=${EPREFIX}
export EASYBUILD_RPATH=1
export EASYBUILD_FILTER_ENV_VARS=LD_LIBRARY_PATH
export EASYBUILD_FILTER_DEPS=$DEPS_TO_FILTER (calculated)
export EASYBUILD_MODULE_EXTENSIONS=1

I also think there is scope for another module for people who want to build their own software (not with EB). We can update the buildenv easyblock to (conditionally) include the rpath wrappers.

[ocaisa]

We will also probably need to place $EPREFIX/bin into PATH

[terjekv]

That's a brilliant idea!

[ocaisa]

At JSC we use

BASH_ENV=/path/to/lmod/lmod/init/bash

so that the module command works in non-interactive shells. If this variable is set in the users session it can break software installation on top of EESSSI with errors that show we are using Lmod from the host rather than EESSI

[ocaisa]

BASH_ENV is an interesting environment variable to leverage so we can automate that jobs submitted to select the right microarchitecture for the node on which the code will run on

[ocaisa]

@bedroge After having a meeting about using the EESSI stack as a baseline installation, I thought a lot about what this means and how you would implement it. In the list of environment variables that are required to build the EESSI stack there are 3 key ones:

export EASYBUILD_SYSROOT=${EPREFIX}
export EASYBUILD_RPATH=1
export EASYBUILD_FILTER_DEPS=$DEPS_TO_FILTER (calculated)

To build on top of EESSI in a consistent way, you need to use the 3 of these. The main problem is that $EPREFIX is (at least currently) a moving target. The way I can see of pinning it down is to stick to one $EPREFIX per toolchain generation, then one can generate a EB hook to automatically chose the right one for you depending on what you are building (if you build something that uses the foss/2020a hierarchy you use the $EPREFIX corresponding to the 2021.03 pilot).

I also thought about being able to retire pilots in EESSI. One way to do this is to make a tarball of what you will remove (e.g., /cvmfs/pilot.eessi-hpc.org/2021.03/), and make that accessible somewhere, you then clean out the directory contents and turn the directory into a variable symlink. If someone want to continue using a retired pilot, they can unpack the tarball locally and set the symlink to point to it.

I'm reworking the install script now to make rebuilding your own module tree easy.

[ocaisa]

In #100 there are the code changes that allow you to rebuild the EESSI module tree according to your preferred naming scheme (has been tested with flat and hierarchical schemes).

[bedroge]

Nice! I also like that idea with the variant symlink for people that want to use an old/removed version. We could dump these in for instance some AWS S3 bucket (and maybe even move them to S3 Glacier at some point).

[terjekv]

Speaking of life-cycle handling of S3 buckets, my testing environment for tarball storage for ingestion to S0 looks like this right now:

resource "aws_s3_bucket" "bucket" {
  provider = eessiaws
  bucket   = "eessi-staging"
  policy   = data.aws_iam_policy_document.public_read_s3.json
  acl      = "private"

  tags = {
    Name = "EESSI staging bucket",    
  }

  lifecycle_rule {
    id      = "layers"
    enabled = true

#    prefix = "/"

    tags = {
      rule      = "layers"
      autoclean = "true"
    }

    transition {
      days          = 30
      storage_class = "STANDARD_IA" # or "ONEZONE_IA"
    }

    transition {
      days          = 60
      storage_class = "GLACIER"
    }

    expiration {
      days = 90
    }
  }
}

S3 is fun. Except for access policies.