[instructions] How to build NMODL on BB5?

[pramodk]

For benchmarking & testing AVX-512 support, provide instructions to build NMODL on BB5.

Note that it's better to not require use of LLVM installation project directories as not everyone has access to same project (e.g. proj16 for George)

[pramodk]

Login to BB5 and you can do:

module purge

module load unstable cmake python-dev flex bison cmake llvm gcc
git clone https://github.com/BlueBrain/nmodl.git
git checkout llvm

mkdir -p nmodl/build
cd nmodl/build

cmake .. -DCMAKE_INSTALL_PREFIX=`pwd`/install -DNMODL_FORMATTING:BOOL=ON -DClangFormat_EXECUTABLE=`which clang-format` -DLLVM_DIR=/gpfs/bbp.cscs.ch/apps/hpc/llvm-install/0521/lib/cmake/llvm  -DNMODL_ENABLE_JIT_EVENT_LISTENERS=ON
make -j8
make install

@georgemitenkov : if you see any issue, let us know!

[pramodk]

Additional info:

• When we login to bbpv1.epfl.ch (i.e. BB5), we are on the login node. On the login node one can build and run simple serial tests. As the login node is shared resource, its not meant for running large simulations or benchmark performance.
• From the login node, one can allocate dedicated compute node as:

salloc --partition=prod --account=proj16 --nodes=1 --time=8:00:00 --exclusive --constraint=clx

This will allocate single node of CascadeLake CPU for 8 hrs. Note that proj16 needs to be replaced with the project that you will be added.

@georgemitenkov : you haven't been added to the specific project. I am checking this internally.

[pramodk]

@georgemitenkov: when time permits, can you check if you can allocate BB5 compute node using following command:

# login to bbpv1.epfl.ch first

# allocate cascade lake 6248 compute node for 8 hours exclusively 
salloc --partition=prod --account=proj7 --nodes=1 --time=8:00:00 --exclusive --constraint=clx

# allocate cascade lake CPU with NVIDIA Volta V100 GPU node for 8 hours exclusively 
salloc --partition=prod --account=proj7 --nodes=1 --time=8:00:00 --exclusive --constraint=volta

After salloc, you are automatically landed on the compute node and you can run any program.

Once you confirm, I will close our internal JIRA ticket.

(Softwares available can be loaded using modules e.g. module load unstable cmake python-dev flex bison cmake llvm intel gcc. I will update additional information for building nmodl here)

[georgemitenkov]

@pramodk I have checked the instructions you have posted here - everything works! Thanks!

[pramodk]

Ok great. And here are instructions that I shared on Slack:

$ cat nmodl_install.sh
mkdir -p nmodl_test && cd nmodl_test

module purge

module load unstable cmake python-dev flex bison cmake llvm gcc
git clone https://github.com/BlueBrain/nmodl.git

mkdir -p nmodl/build
cd nmodl/build
git checkout llvm

cmake .. -DCMAKE_INSTALL_PREFIX=`pwd`/install -DNMODL_FORMATTING:BOOL=ON -DClangFormat_EXECUTABLE=`which clang-format` -DLLVM_DIR=/gpfs/bbp.cscs.ch/apps/hpc/llvm-install/0521/lib/cmake/llvm  -DNMODL_ENABLE_JIT_EVENT_LISTENERS=ON
make -j8
make install

To use Intel SVML:

module load unstable intel
./bin/nmodl  ../test.mod llvm --ir --vector-width 8 --veclib SVML \
  --opt benchmark --run --instance-size 20000000 --repeat 5  \
  --libs /gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/compilers/2021-01-06/linux-rhel7-x86_64/gcc-4.8.5/intel-20.0.2-ilowey/compilers_and_libraries_2020.2.254/linux/compiler/lib/intel64_lin/libsvml.so \
--opt-level-ir 3 --opt-level-codegen  3  --backend default

This will be useful when you will need to re-run benchmarking results that Giacomo has been preparing in #666.

As we have working instructions, I will close this.