diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_1.txt new file mode 100644 index 00000000000..0c1aacb59da --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_1.txt @@ -0,0 +1,41 @@ +MATLAB + note + To run a MATLAB program on the HPC-UGent infrastructure you must compile it first, + because the MATLAB license server is not accessible from cluster workernodes + (except for the interactive debug cluster). + Compiling MATLAB programs is only possible on the interactive debug cluster, + not on the HPC-UGent login nodes where resource limits w.r.t. memory and max. number of progress are too strict. +Why is the MATLAB compiler required? +The main reason behind this alternative way of using MATLAB is +licensing: only a limited number of MATLAB sessions can be active at the +same time. However, once the MATLAB program is compiled using the MATLAB +compiler, the resulting stand-alone executable can be run without +needing to contact the license server. +Note that a license is required for the MATLAB Compiler, see +https://nl.mathworks.com/help/compiler/index.html. If the mcc +command is provided by the MATLAB installation you are using, the MATLAB +compiler can be used as explained below. +Only a limited amount of MATLAB sessions can be active at the same time +because there are only a limited amount of MATLAB research licenses +available on the UGent MATLAB license server. If each job would need a +license, licenses would quickly run out. +How to compile MATLAB code +Compiling MATLAB code can only be done from the login nodes, because +only login nodes can access the MATLAB license server, workernodes on +clusters cannot. +To access the MATLAB compiler, the MATLAB module should be loaded +first. Make sure you are using the same MATLAB version to compile and +to run the compiled MATLAB program. +$ module avail MATLAB/ +----------------------/apps/gent/RHEL8/zen2-ib/modules/all---------------------- + MATLAB/2021b MATLAB/2022b-r5 (D) +$ module load MATLAB/2021b +After loading the MATLAB module, the mcc command can be used. To get +help on mcc, you can run mcc -?. +To compile a standalone application, the -m flag is used (the -v +flag means verbose output). To show how mcc can be used, we use the +magicsquare example that comes with MATLAB. +First, we copy the magicsquare.m example that comes with MATLAB to +example.m: +cp $EBROOTMATLAB/extern/examples/compiler/magicsquare.m example.m +To compile a MATLAB program, use mcc -mv: diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_1_metadata.json new file mode 100644 index 00000000000..88cf1ca964c --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "MATLAB", + "subtitle": "How-to-compile-MATLAB-code", + "source_file": "../../mkdocs/docs/HPC/MATLAB.md", + "title_depth": 2, + "directory": "MATLAB", + "links": { + "0": "https://docs.hpc.ugent.be/interactive_debug" + }, + "parent_title": "", + "previous_title": null, + "next_title": "MATLAB_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/MATLAB/#how-to-compile-matlab-code" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_2.txt new file mode 100644 index 00000000000..e67e73c81b7 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_2.txt @@ -0,0 +1,47 @@ +mcc -mv example.m +Opening log file: /user/home/gent/vsc400/vsc40000/java.log.34090 +Compiler version: 8.3 (R2021b) +Dependency analysis by REQUIREMENTS. +Parsing file "/user/home/gent/vsc400/vsc40000/example.m" + (Referenced from: "Compiler Command Line"). +Deleting 0 temporary MEX authorization files. +Generating file "/user/home/gent/vsc400/vsc40000/readme.txt". +Generating file "run\_example.sh". +Libraries +To compile a MATLAB program that needs a library, you can use the +-I library_path flag. This will tell the compiler to also look for +files in library_path. +It's also possible to use the -a path flag. That will result in all +files under the path getting added to the final executable. +For example, the command mcc -mv example.m -I examplelib -a datafiles +will compile example.m with the MATLAB files in examplelib, and will +include all files in the datafiles directory in the binary it +produces. +Memory issues during compilation +If you are seeing Java memory issues during the compilation of your +MATLAB program on the login nodes, consider tweaking the default maximum +heap size (128M) of Java using the _JAVA_OPTIONS environment variable +with: +export _JAVA_OPTIONS="-Xmx64M" +The MATLAB compiler spawns multiple Java processes. Because of the +default memory limits that are in effect on the login nodes, this might +lead to a crash of the compiler if it's trying to create to many Java +processes. If we lower the heap size, more Java processes will be able +to fit in memory. +Another possible issue is that the heap size is too small. This could +result in errors like: +Error: Out of memory +A possible solution to this is by setting the maximum heap size to be +bigger: +export _JAVA_OPTIONS="-Xmx512M" +Multithreading +MATLAB can only use the cores in a single workernode (unless the +Distributed Computing toolbox is used, see +https://nl.mathworks.com/products/distriben.html). +The amount of workers used by MATLAB for the parallel toolbox can be +controlled via the parpool function: parpool(16) will use 16 +workers. It's best to specify the amount of workers, because otherwise +you might not harness the full compute power available (if you have too +few workers), or you might negatively impact performance (if you have +too many workers). By default, MATLAB uses a fixed number of workers +(12). diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_2_metadata.json new file mode 100644 index 00000000000..afa4b12f574 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_2_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "MATLAB", + "subtitle": "Multithreading", + "source_file": "../../mkdocs/docs/HPC/MATLAB.md", + "title_depth": 2, + "directory": "MATLAB", + "parent_title": "", + "previous_title": "MATLAB_paragraph_1", + "next_title": "MATLAB_paragraph_3", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/MATLAB/#multithreading" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_3.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_3.txt new file mode 100644 index 00000000000..1d2c6f23971 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_3.txt @@ -0,0 +1,55 @@ +You should use a number of workers that is equal to the number of cores +you requested when submitting your job script (the ppn value, see Generic resource requirements). +You can determine the right number of workers to use via the following +code snippet in your MATLAB program: +% specify the right number of workers (as many as there are cores available in the job) when creating the parpool +c = parcluster('local') +pool = parpool(c.NumWorkers) +See also [the parpool +documentation](https://nl.mathworks.com/help/distcomp/parpool.html). +Java output logs +Each time MATLAB is executed, it generates a Java log file in the users +home directory. The output log directory can be changed using: +MATLAB_LOG_DIR= +where is the name of the desired output directory. To +create and use a temporary directory for these logs: +# create unique temporary directory in $TMPDIR (or /tmp/$USER if +$TMPDIR is not defined) +# instruct MATLAB to use this directory for log files by setting $MATLAB_LOG_DIR +$ export MATLAB_LOG_DIR=$ (mktemp -d -p $TMPDIR:-/tmp/$USER) +You should remove the directory at the end of your job script: +rm -rf $MATLAB_LOG_DIR +Cache location +When running, MATLAB will use a cache for performance reasons. This +location and size of this cache can be changed through the +MCR_CACHE_ROOT and MCR_CACHE_SIZE environment variables. +The snippet below would set the maximum cache size to 1024MB and the +location to /tmp/testdirectory. +export MATLAB_CACHE_ROOT=/tmp/testdirectory +export MATLAB_CACHE_SIZE=1024M +So when MATLAB is running, it can fill up to 1024MB of cache in +/tmp/testdirectory. +MATLAB job script +All of the tweaks needed to get MATLAB working have been implemented in +an example job script. This job script is also available on the HPC. + +#!/bin/bash +#PBS -l nodes=1:ppn=1 +#PBS -l walltime=1:0:0 +# +# Example (single-core) MATLAB job script +# +# make sure the MATLAB version matches with the one used to compile the MATLAB program! +module load MATLAB/2021b +# use temporary directory (not $HOME) for (mostly useless) MATLAB log files +# subdir in $TMPDIR (if defined, or /tmp otherwise) +export MATLAB_LOG_DIR=$(mktemp -d -p ${TMPDIR:-/tmp}) +# configure MATLAB Compiler Runtime cache location & size (1GB) +# use a temporary directory in /dev/shm (i.e. in memory) for performance reasons +export MCR_CACHE_ROOT=$(mktemp -d -p /dev/shm) +export MCR_CACHE_SIZE=1024MB +# change to directory where job script was submitted from +cd $PBS_O_WORKDIR +# run compiled example MATLAB program 'example', provide '5' as input argument to the program +# $EBROOTMATLAB points to MATLAB installation directory +./run_example.sh $EBROOTMATLAB 5 diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_3_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_3_metadata.json new file mode 100644 index 00000000000..8bbfc6b0df3 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_3_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "MATLAB", + "subtitle": "MATLAB-job-script", + "source_file": "../../mkdocs/docs/HPC/MATLAB.md", + "title_depth": 2, + "directory": "MATLAB", + "links": { + "0": "https://docs.hpc.ugent.be/running_batch_jobs/#generic-resource-requirements" + }, + "parent_title": "", + "previous_title": "MATLAB_paragraph_2", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/MATLAB/#matlab-job-script" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/index/index_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/index/index_paragraph_1.txt new file mode 100644 index 00000000000..5d8dfad70c3 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/index/index_paragraph_1.txt @@ -0,0 +1,17 @@ +--- +hide: "toc" +--- + Welcome to the HPC-UGent documentation +--- +Use the menu on the left to navigate, or use the search box on the top right. +You are viewing documentation intended for people using *. +Use the OS dropdown in the top bar to switch to a different operating system. +Quick links +- Getting Started | Getting Access +- Recording of HPC-UGent intro +- Linux Tutorial +- Hardware overview +- FAQ | Troubleshooting | Best practices | Known issues + +If you find any problems in this documentation, please report them by mail to or open a pull request. +If you still have any questions, you can contact the HPC-UGent team. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/index/index_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/index/index_paragraph_1_metadata.json new file mode 100644 index 00000000000..de0f9823689 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/index/index_paragraph_1_metadata.json @@ -0,0 +1,25 @@ +{ + "main_title": "index", + "subtitle": "Welcome-to-the-HPC-UGent-documentation", + "source_file": "../../mkdocs/docs/HPC/index.md", + "title_depth": 1, + "directory": "index", + "links": { + "0": "https://docs.hpc.ugent.be/getting_started", + "1": "https://docs.hpc.ugent.be/account", + "2": "https://www.ugent.be/hpc/en/training/introhpcugent-recording", + "3": "https://docs.hpc.ugent.be/linux-tutorial", + "4": "https://www.ugent.be/hpc/en/infrastructure", + "5": "https://docs.hpc.ugent.be/FAQ", + "6": "https://docs.hpc.ugent.be/troubleshooting", + "7": "https://docs.hpc.ugent.be/best_practices", + "8": "https://docs.hpc.ugent.be/known_issues", + "9": "https://github.com/hpcugent/vsc_user_docs", + "10": "https://www.ugent.be/hpc/en/support" + }, + "parent_title": "", + "previous_title": null, + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/interactive_debug/interactive_debug_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/interactive_debug/interactive_debug_paragraph_1.txt new file mode 100644 index 00000000000..1b04b6db10e --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/interactive_debug/interactive_debug_paragraph_1.txt @@ -0,0 +1,44 @@ +Interactive and debug cluster +Purpose +The purpose of this cluster is to give the user an environment where +there should be no waiting in the queue to get access to a limited +number of resources. This environment allows a user to immediately +start working, and is the ideal place for interactive work such as +development, debugging and light production workloads (typically +sufficient for training and/or courses). +This environment should be seen as an extension or even replacement of the login nodes, +instead of a dedicated compute resource. The interactive cluster is +overcommitted, which means that more CPU cores can be requested for +jobs than physically exist in the cluster. Obviously, the performance of +this cluster heavily depends on the workloads and the actual overcommit +usage. Be aware that jobs can slow down or speed up during their +execution. +Due to the restrictions and sharing of the CPU resources (see +section Restrictions and overcommit factor) jobs on this cluster +should normally start more or less immediately. The tradeoff is that +performance must not be an issue for the submitted jobs. This means that +typical workloads for this cluster should be limited to: +- Interactive jobs (see + chapter Running interactive jobs) +- Cluster desktop sessions (see + chapter Using the HPC-UGent web portal) +- Jobs requiring few resources +- Debugging programs +- Testing and debugging job scripts +Submitting jobs +To submit jobs to the HPC-UGent interactive and debug cluster nicknamed +donphan, first use: +module swap cluster/donphan +Then use the familiar qsub, qstat, etc. commands (see +chapter Running batch jobs). +Restrictions and overcommit factor +Some limits are in place for this cluster: +- each user may have at most 5 jobs in the queue (both running and + waiting to run); +- at most 3 jobs per user can be running at the same time; +- running jobs may allocate no more than 8 CPU cores and no more than + 27200 MiB of memory in total, per user; +In addition, the cluster has an overcommit factor of 6. This means that +6 times more cores can be allocated than physically exist. +Simultaneously, the default memory per core is 6 times less than what +would be available on a non-overcommitted cluster. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/interactive_debug/interactive_debug_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/interactive_debug/interactive_debug_paragraph_1_metadata.json new file mode 100644 index 00000000000..678b7531686 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/interactive_debug/interactive_debug_paragraph_1_metadata.json @@ -0,0 +1,18 @@ +{ + "main_title": "interactive_debug", + "subtitle": "Restrictions-and-overcommit-factor", + "source_file": "../../mkdocs/docs/HPC/interactive_debug.md", + "title_depth": 3, + "directory": "interactive_debug", + "links": { + "0": "https://docs.hpc.ugent.be/interactive_debug/#restrictions-and-overcommit-factor", + "1": "https://docs.hpc.ugent.be/running_interactive_jobs/#running-interactive-jobs", + "2": "https://docs.hpc.ugent.be/web_portal/#using-the-hpc-ugent-web-portal", + "3": "https://docs.hpc.ugent.be/running_batch_jobs/#running-batch-jobs" + }, + "parent_title": "", + "previous_title": null, + "next_title": "interactive_debug_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/interactive_debug/#restrictions-and-overcommit-factor" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/interactive_debug/interactive_debug_paragraph_2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/interactive_debug/interactive_debug_paragraph_2.txt new file mode 100644 index 00000000000..b40fd4388ec --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/interactive_debug/interactive_debug_paragraph_2.txt @@ -0,0 +1,11 @@ +Please note that based on the (historical) workload of the interactive +and debug cluster, the above restrictions and the overcommitment ratio +might change without prior notice. +Shared GPUs +Each node in the donphan cluster has a relatively small GPU that is shared between all jobs. +This means that you don't need to reserve it and thus possibly wait for it. +But this also has a downside for performance and security: jobs might be competing for the same GPU resources (cores, memory or encoders) without +any preset fairshare and there is no guarantee one job cannot access another job's memory +(as opposed to having reserved GPUs in the GPU clusters). +All software should behave the same as on the dedicated GPU clusters (e.g. using CUDA or OpenGL acceleration +from a cluster desktop via the webportal). \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/interactive_debug/interactive_debug_paragraph_2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/interactive_debug/interactive_debug_paragraph_2_metadata.json new file mode 100644 index 00000000000..469fbe3f801 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/interactive_debug/interactive_debug_paragraph_2_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "interactive_debug", + "subtitle": "Shared-GPUs", + "source_file": "../../mkdocs/docs/HPC/interactive_debug.md", + "title_depth": 3, + "directory": "interactive_debug", + "parent_title": "", + "previous_title": "interactive_debug_paragraph_1", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/interactive_debug/#shared-gpus" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_1.txt new file mode 100644 index 00000000000..f796f2b19a2 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_1.txt @@ -0,0 +1,42 @@ +Introduction to HPC +What is HPC? +"High Performance Computing" (HPC) is computing on a "Supercomputer", a computer with at the +frontline of contemporary processing capacity -- particularly speed of +calculation and available memory. +While the supercomputers in the early days (around 1970) used only a few +processors, in the 1990s machines with thousands of processors began to +appear and, by the end of the 20th century, massively parallel +supercomputers with tens of thousands of "off-the-shelf" processors were +the norm. A large number of dedicated processors are placed in close +proximity to each other in a computer cluster. +A computer cluster consists of a set of loosely or tightly connected computers that work +together so that in many respects they can be viewed as a single system. +The components of a cluster are usually connected to each other through +fast local area networks ("LAN") with each node (computer used as a +server) running its own instance of an operating system. Computer +clusters emerged as a result of convergence of a number of computing +trends including the availability of low cost microprocessors, +high-speed networks, and software for high performance distributed +computing. +Compute clusters are usually deployed to improve performance and +availability over that of a single computer, while typically being more +cost-effective than single computers of comparable speed or +availability. +Supercomputers play an important role in the field of computational +science, and are used for a wide range of computationally intensive +tasks in various fields, including quantum mechanics, weather +forecasting, climate research, oil and gas exploration, molecular +modelling (computing the structures and properties of chemical +compounds, biological macromolecules, polymers, and crystals), and +physical simulations (such as simulations of the early moments of the +universe, airplane and spacecraft aerodynamics, the detonation of +nuclear weapons, and nuclear fusion). [^1] +What is the HPC-UGent infrastructure? +The HPC is a collection of computers with AMD and/or Intel CPUs, running a +Linux operating system, shaped like pizza boxes and stored above and +next to each other in racks, interconnected with copper and fiber +cables. Their number crunching power is (presently) measured in hundreds +of billions of floating point operations (gigaflops) and even in +teraflops. +The HPC-UGent infrastructure relies on parallel-processing technology to offer UGent researchers an +extremely fast solution for all their data processing needs. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_1_metadata.json new file mode 100644 index 00000000000..2ec8e0ef5c8 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_1_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "introduction", + "subtitle": "What-is-the-HPC-UGent-infrastructure", + "source_file": "../../mkdocs/docs/HPC/introduction.md", + "title_depth": 2, + "directory": "introduction", + "parent_title": "", + "previous_title": null, + "next_title": "introduction_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/introduction/#what-is-the-hpc-ugent-infrastructure" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_2.txt new file mode 100644 index 00000000000..6839c6b8f7d --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_2.txt @@ -0,0 +1,50 @@ +The HPC currently consists of: +Two tools perform the Job management and job scheduling: +1. TORQUE: a resource manager (based on PBS); +2. Moab: job scheduler and management tools. +What the HPC infrastucture is not +The HPC infrastructure is not a magic computer that automatically: +1. runs your PC-applications much faster for bigger problems; +2. develops your applications; +3. solves your bugs; +4. does your thinking; +5. ... +6. allows you to play games even faster. +The HPC does not replace your desktop computer. +Is the HPC a solution for my computational needs? +Batch or interactive mode? +Typically, the strength of a supercomputer comes from its ability to run +a huge number of programs (i.e., executables) in parallel without any +user interaction in real time. This is what is called "running in batch +mode". +It is also possible to run programs at the HPC, which require user +interaction. (pushing buttons, entering input data, etc.). Although +technically possible, the use of the HPC might not always be the best and +smartest option to run those interactive programs. Each time some user +interaction is needed, the computer will wait for user input. The +available computer resources (CPU, storage, network, etc.) might not be +optimally used in those cases. A more in-depth analysis with the HPC staff +can unveil whether the HPC is the desired solution to run interactive +programs. Interactive mode is typically only useful for creating quick +visualisations of your data without having to copy your data to your +desktop and back. +What are cores, processors and nodes? +In this manual, the terms core, processor and node will be frequently +used, so it's useful to understand what they are. +Modern servers, also referred to as *(worker)nodes* in the context of +HPC, include one or more sockets, each housing a multi-core +processor (next to memory, disk(s), network cards, ...). A modern +processor consists of multiple CPUs or cores that are used to +execute computations. +Parallel or sequential programs? +Parallel programs +Parallel computing is a form of computation in which many calculations are carried out +simultaneously. They are based on the principle that large problems can +often be divided into smaller ones, which are then solved concurrently +("in parallel"). +Parallel computers can be roughly classified according to the level at +which the hardware supports parallelism, with multicore computers having +multiple processing elements within a single machine, while clusters use +multiple computers to work on the same task. Parallel computing has +become the dominant computer architecture, mainly in the form of +multicore processors. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_2_metadata.json new file mode 100644 index 00000000000..03764bebb6c --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_2_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "introduction", + "subtitle": "Parallel-programs", + "source_file": "../../mkdocs/docs/HPC/introduction.md", + "title_depth": 4, + "directory": "introduction", + "parent_title": "", + "previous_title": "introduction_paragraph_1", + "next_title": "introduction_paragraph_3", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/introduction/#parallel-programs" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_3.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_3.txt new file mode 100644 index 00000000000..57fe9718534 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_3.txt @@ -0,0 +1,34 @@ +The two parallel programming paradigms most used in HPC are: +- OpenMP for shared memory systems (multithreading): on multiple cores + of a single node +- MPI for distributed memory systems (multiprocessing): on multiple + nodes +Parallel programs are more difficult to write than sequential ones, because concurrency +introduces several new classes of potential software bugs, of which race +conditions are the most common. Communication and synchronisation +between the different subtasks are typically some of the greatest +obstacles to getting good parallel program performance. +Sequential programs +Sequential software does not do calculations in parallel, i.e., it only +uses one single core of a single workernode. It does not become faster by just throwing more cores at it: it can only use one +core. +It is perfectly possible to also run purely sequential programs on the HPC. +Running your sequential programs on the most modern and fastest +computers in the HPC can save you a lot of time. But it also might be +possible to run multiple instances of your program (e.g., with different +input parameters) on the HPC, in order to solve one overall problem (e.g., +to perform a parameter sweep). This is another form of running your +sequential programs in parallel. +What programming languages can I use? +You can use any programming language, any software package and any +library provided it has a version that runs on Linux, specifically, on +the version of Linux that is installed on the compute nodes, RHEL 8.8 (accelgor, doduo, donphan, gallade, joltik, skitty). +For the most common programming languages, a compiler is available on RHEL 8.8 (accelgor, doduo, donphan, gallade, joltik, skitty). Supported and common +programming languages on the HPC are C/C++, FORTRAN, Java, Perl, Python, +MATLAB, R, etc. +Supported and commonly used compilers are GCC and Intel. +Additional software can be installed "on demand". Please contact the HPC +staff to see whether the HPC can handle your specific requirements. +What operating systems can I use? +All nodes in the HPC cluster run under RHEL 8.8 (accelgor, doduo, donphan, gallade, joltik, skitty), which is a specific version of Red Hat Enterprise Linux. +This means that all programs (executables) should be compiled for RHEL 8.8 (accelgor, doduo, donphan, gallade, joltik, skitty). diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_3_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_3_metadata.json new file mode 100644 index 00000000000..0ce53198cd2 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_3_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "introduction", + "subtitle": "What-operating-systems-can-I-use", + "source_file": "../../mkdocs/docs/HPC/introduction.md", + "title_depth": 3, + "directory": "introduction", + "parent_title": "", + "previous_title": "introduction_paragraph_2", + "next_title": "introduction_paragraph_4", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/introduction/#what-operating-systems-can-i-use" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_4.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_4.txt new file mode 100644 index 00000000000..6bb34646152 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_4.txt @@ -0,0 +1,26 @@ +Users can connect from any computer in the UGent network to the HPC, regardless +of the Operating System that they are using on their personal computer. +Users can use any of the common Operating Systems (such as Windows, +macOS or any version of Linux/Unix/BSD) and run and control their +programs on the HPC. +A user does not need to have prior knowledge about Linux; all of the +required knowledge is explained in this tutorial. +What does a typical workflow look like? +A typical workflow looks like: +1. Connect to the login nodes with SSH (see First Time connection to the HPC infrastructure) +2. Transfer your files to the cluster (see Transfer Files to/from the HPC) +3. Optional: compile your code and test it (for compiling, see Compiling and testing your software on the HPC) +4. Create a job script and submit your job (see Running batch jobs) +5. Get some coffee and be patient: + 1. Your job gets into the queue + 2. Your job gets executed + 3. Your job finishes +6. Study the results generated by your jobs, either on the cluster or + after downloading them locally. +What is the next step? +When you think that the HPC is a useful tool to support your computational +needs, we encourage you to acquire a VSC-account (as explained in Getting a HPC Account), +read Connecting to the HPC infrastructure, "Setting up the environment", and explore +chapters Running interactive jobs to Fine-tuning Job Specifications which will help you to transfer and run your programs on the HPC cluster. +Do not hesitate to contact the HPC staff for any help. +[^1]: Wikipedia: http://en.wikipedia.org/wiki/Supercomputer \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_4_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_4_metadata.json new file mode 100644 index 00000000000..a05c924b7e7 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_4_metadata.json @@ -0,0 +1,22 @@ +{ + "main_title": "introduction", + "subtitle": "What-is-the-next-step", + "source_file": "../../mkdocs/docs/HPC/introduction.md", + "title_depth": 3, + "directory": "introduction", + "links": { + "0": "https://docs.hpc.ugent.be/connecting/#first-time-connection-to-the-hpc-infrastructure", + "1": "https://docs.hpc.ugent.be/connecting/#transfer-files-tofrom-the-hpc", + "2": "https://docs.hpc.ugent.be/", + "3": "https://docs.hpc.ugent.be/", + "4": "https://docs.hpc.ugent.be/account", + "5": "https://docs.hpc.ugent.be/connecting", + "6": "https://docs.hpc.ugent.be/", + "7": "https://docs.hpc.ugent.be/" + }, + "parent_title": "", + "previous_title": "introduction_paragraph_3", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/introduction/#what-is-the-next-step" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jobscript_examples/jobscript_examples_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jobscript_examples/jobscript_examples_paragraph_1.txt new file mode 100644 index 00000000000..8ae0b56d545 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jobscript_examples/jobscript_examples_paragraph_1.txt @@ -0,0 +1,43 @@ +Job script examples +Simple job script template +This is a template for a job script, with commonly used parameters. +The basic parameters should always be used. Some notes on the situational parameters: +* -l mem: If no memory parameter is given, the job gets access to an amount of + memory proportional to the amount of cores requested. + See also: Job failed: SEGV Segmentation fault +* -m/-M: the -m option will send emails to your email address registerd with VSC. + Only if you want emails at some other address, you should use the -M option. +* Replace the "-placeholder text-" with real entries. + This notation is used to ensure qsub rejects invalid options. +* To use a situational parameter, remove one '#' at the beginning of the line. +#!/bin/bash +# Basic parameters +#PBS -N jobname ## Job name +#PBS -l nodes=1:ppn=2 ## 1 node, 2 processors per node (ppn=all to get a full node) +#PBS -l walltime=01:00:00 ## Max time your job will run (no more than 72:00:00) +# Situational parameters: remove one '#' at the front to use +##PBS -l gpus=1 ## GPU amount (only on accelgor or joltik) +##PBS -l mem=32gb ## If not used, memory will be available proportional to the max amount +##PBS -m abe ## Email notifications (abe=aborted, begin and end) +##PBS -M -email_address- ## ONLY if you want to use a different email than your VSC address +##PBS -A -project- ## Project name when credits are required (only Tier 1) +##PBS -o -filename- ## Output log +##PBS -e -filename- ## Error log +module load [module] +module load [module] +cd $PBS_O_WORKDIR # Change working directory to the location where the job was submmitted +[commands] +Single-core job +Here's an example of a single-core job script: +#!/bin/bash +#PBS -N count_example ## job name +#PBS -l nodes=1:ppn=1 ## single-node job, single core +#PBS -l walltime=2:00:00 ## max. 2h of wall time +module load Python/3.6.4-intel-2018a +# copy input data from location where job was submitted from +cp $PBS_O_WORKDIR/input.txt $TMPDIR +# go to temporary working directory (on local disk) & run +cd $TMPDIR +python -c "print(len(open('input.txt').read()))" > output.txt +# copy back output data, ensure unique filename using $PBS_JOBID +cp output.txt $VSC_DATA/output_${PBS_JOBID}.txt diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jobscript_examples/jobscript_examples_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jobscript_examples/jobscript_examples_paragraph_1_metadata.json new file mode 100644 index 00000000000..68b42f01550 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jobscript_examples/jobscript_examples_paragraph_1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "jobscript_examples", + "subtitle": "Single-core-job", + "source_file": "../../mkdocs/docs/HPC/jobscript_examples.md", + "title_depth": 2, + "directory": "jobscript_examples", + "links": { + "0": "https://docs.hpc.ugent.be/FAQ/#job-failed-segv-segmentation-fault" + }, + "parent_title": "", + "previous_title": null, + "next_title": "jobscript_examples_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/jobscript_examples/#single-core-job" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jobscript_examples/jobscript_examples_paragraph_2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jobscript_examples/jobscript_examples_paragraph_2.txt new file mode 100644 index 00000000000..b7bc7f23eaa --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jobscript_examples/jobscript_examples_paragraph_2.txt @@ -0,0 +1,48 @@ +1. Using #PBS header lines, we specify the resource requirements for + the job, see Apendix B for a list of these options. +2. A module for Python 3.6 is loaded, see also section Modules. +3. We stage the data in: the file input.txt is copied into the + "working" directory, see chapter Running jobs with input/output data. +4. The main part of the script runs a small Python program that counts + the number of characters in the provided input file input.txt. +5. We stage the results out: the output file output.txt is copied + from the "working directory" ($TMPDIR|) to a unique directory in + $VSC_DATA. For a list of possible storage locations, see subsection Pre-defined user directories. +Multi-core job +Here's an example of a multi-core job script that uses mympirun: +#!/bin/bash +#PBS -N mpi_hello ## job name +#PBS -l nodes=2:ppn=all ## 2 nodes, all cores per node +#PBS -l walltime=2:00:00 ## max. 2h of wall time +module load intel/2017b +module load vsc-mympirun ## We don't use a version here, this is on purpose +# go to working directory, compile and run MPI hello world +cd $PBS_O_WORKDIR +mpicc mpi_hello.c -o mpi_hello +mympirun ./mpi_hello +An example MPI hello world program can be downloaded from +https://github.com/hpcugent/vsc-mympirun/blob/master/testscripts/mpi_helloworld.c. +Running a command with a maximum time limit +If you want to run a job, but you are not sure it will finish before the +job runs out of walltime and you want to copy data back before, you have +to stop the main command before the walltime runs out and copy the data +back. +This can be done with the timeout command. This command sets a limit +of time a program can run for, and when this limit is exceeded, it kills +the program. Here's an example job script using timeout: +#!/bin/bash +#PBS -N timeout_example +#PBS -l nodes=1:ppn=1 ## single-node job, single core +#PBS -l walltime=2:00:00 ## max. 2h of wall time +# go to temporary working directory (on local disk) +cd $TMPDIR +# This command will take too long (1400 minutes is longer than our walltime) +# $PBS_O_WORKDIR/example_program.sh 1400 output.txt +# So we put it after a timeout command +# We have a total of 120 minutes (2 x 60) and we instruct the script to run for +# 100 minutes, but timeout after 90 minute, +# so we have 30 minutes left to copy files back. This should +# be more than enough. +timeout -s SIGKILL 90m $PBS_O_WORKDIR/example_program.sh 100 output.txt +# copy back output data, ensure unique filename using $PBS_JOBID +cp output.txt $VSC_DATA/output_${PBS_JOBID}.txt diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jobscript_examples/jobscript_examples_paragraph_2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jobscript_examples/jobscript_examples_paragraph_2_metadata.json new file mode 100644 index 00000000000..fe6c832d111 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jobscript_examples/jobscript_examples_paragraph_2_metadata.json @@ -0,0 +1,18 @@ +{ + "main_title": "jobscript_examples", + "subtitle": "Running-a-command-with-a-maximum-time-limit", + "source_file": "../../mkdocs/docs/HPC/jobscript_examples.md", + "title_depth": 2, + "directory": "jobscript_examples", + "links": { + "0": "https://docs.hpc.ugent.be/torque_options", + "1": "https://docs.hpc.ugent.be/running_batch_jobs/#modules", + "2": "https://docs.hpc.ugent.be/running_jobs_with_input_output_data", + "3": "https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#pre-defined-user-directories" + }, + "parent_title": "", + "previous_title": "jobscript_examples_paragraph_1", + "next_title": "jobscript_examples_paragraph_3", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/jobscript_examples/#running-a-command-with-a-maximum-time-limit" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jobscript_examples/jobscript_examples_paragraph_3.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jobscript_examples/jobscript_examples_paragraph_3.txt new file mode 100644 index 00000000000..9d3670dc102 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jobscript_examples/jobscript_examples_paragraph_3.txt @@ -0,0 +1,13 @@ +The example program used in this script is a dummy script that simply +sleeps a specified amount of minutes: +#!/bin/bash +# This is an example program +# It takes two arguments: a number of times to loop and a file to write to +# In total, it will run for (the number of times to loop) minutes +if [ $# -ne 2 ]; then + echo "Usage: ./example_program amount filename" && exit 1 +fi +for ((i = 0; i < $1; i++ )); do + echo "${i} => $(date)" >> $2 + sleep 60 +done diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jobscript_examples/jobscript_examples_paragraph_3_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jobscript_examples/jobscript_examples_paragraph_3_metadata.json new file mode 100644 index 00000000000..1d1fc47b0ba --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jobscript_examples/jobscript_examples_paragraph_3_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "jobscript_examples", + "subtitle": "Running-a-command-with-a-maximum-time-limit", + "source_file": "../../mkdocs/docs/HPC/jobscript_examples.md", + "title_depth": 2, + "directory": "jobscript_examples", + "parent_title": "", + "previous_title": "jobscript_examples_paragraph_2", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/jobscript_examples/#running-a-command-with-a-maximum-time-limit" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jupyter/jupyter_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jupyter/jupyter_paragraph_1.txt new file mode 100644 index 00000000000..e75d1f3f59f --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jupyter/jupyter_paragraph_1.txt @@ -0,0 +1,34 @@ +Jupyter notebook +What is a Jupyter notebook +A Jupyter notebook is an interactive, web-based environment that allows you to create documents that contain live code, equations, visualizations, and plaintext. The code blocks in these documents can be used to write Python, Java, R and Julia code, among others. The combination of code executions with text and visual outputs make it a useful tool for data analysis, machine learning and educational purposes. +Using Jupyter Notebooks on the HPC +Launching a notebook using the web portal +Through the HPC-UGent web portal you can easily start a Jupyter notebook on a workernode, via the Jupyter Notebook button under the Interactive Apps menu item. + + +After starting the Jupyter notebook using the Launch button, you will see it being added in state Queued in the overview of interactive sessions (see My Interactive Sessions menu item): + + +When your job hosting the Jupyter notebook starts running, the status will first change the Starting: + + +and eventually the status will change to Running, and you will be able to connect to the Jupyter environment using the blue Connect to Jupyter button: + + +This will launch the Jupyter environment in a new browser tab, where you can open an existing notebook by navigating to the directory where it is located and clicking it. You can also create a new notebook by clicking on File>New>Notebook: + + +Using extra Python packages +A number of Python packages are readily available in modules on the HPC. To illustrate how to use them in a Jupyter notebook, we will make use of an example where we want to use numpy in our notebook. +The first thing we need to do is finding the modules that contain our package of choice. For numpy, this would be the SciPy-bundle modules. +To find the appropriate modules, it is recommended to use the shell within the web portal under Clusters>>_login Shell Access. + + +We can see all available versions of the SciPy module by using module avail SciPy-bundle: +$ module avail SciPy-bundle +------------------ /apps/gent/RHEL8/zen2-ib/modules/all ------------------ + SciPy-bundle/2022.05-foss-2022a SciPy-bundle/2023.11-gfbf-2023b (D) + SciPy-bundle/2023.07-gfbf-2023a + Where: + D: Default Module +... diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jupyter/jupyter_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jupyter/jupyter_paragraph_1_metadata.json new file mode 100644 index 00000000000..76b1aadec09 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jupyter/jupyter_paragraph_1_metadata.json @@ -0,0 +1,16 @@ +{ + "main_title": "jupyter", + "subtitle": "Using-extra-Python-packages", + "source_file": "../../mkdocs/docs/HPC/jupyter.md", + "title_depth": 3, + "directory": "jupyter", + "links": { + "0": "https://jupyter.org/", + "1": "https://docs.hpc.ugent.be/web_portal" + }, + "parent_title": "", + "previous_title": null, + "next_title": "jupyter_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/jupyter/#using-extra-python-packages" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jupyter/jupyter_paragraph_2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jupyter/jupyter_paragraph_2.txt new file mode 100644 index 00000000000..8e678844d3c --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jupyter/jupyter_paragraph_2.txt @@ -0,0 +1,16 @@ +Not all modules will work for every notebook, we need to use the one that uses the same toolchain as the notebook we want to launch. To find that toolchain, we can look at the JupyterNotebook version field when creating a notebook. In our example 7.2.0 is the version of the notebook and GCCcore/13.2.0 is the toolchain used. + + +Module names include the toolchain that was used to install the module (for example gfbf-2023b in SciPy-bundle/2023.11-gfbf-2023b means that that module uses the toolchain gfbf/2023). To see which modules are compatible with each other, you can check the table on the page about Module conflicts. Another way to find out which GCCcore subtoolchain goes with the particular toolchain of the module (such as gfbf/2023b) is to use module show. In particular using module show | grep GCC (before the module has been loaded) will return this GCCcore version. +$ module show gfbf/2023b | grep GCC +GNU Compiler Collection (GCC) based compiler toolchain, including +whatis("Description: GNU Compiler Collection (GCC) based compiler toolchain, including +load("GCC/13.2.0") +load("FlexiBLAS/3.3.1-GCC-13.2.0") +load("FFTW/3.3.10-GCC-13.2.0") +The toolchain used can then for example be found within the line load("GCC/13.2.0") and the included Python packages under the line Included extensions. +It is also recommended to doublecheck the compatibility of the Jupyter notebook version and the extra modules by loading them all in a shell environment. +To do so, find the module containing the correct Jupyter notebook version (for our example case this is JupyterNotebook/7.2.0-GCCcore-13.2.0) and then use module load for every module as follows: +$ module load JupyterNotebook/7.2.0-GCCcore-13.2.0 +$ module load SciPy-bundle/2023.11-gfbf-2023b +This throws no errors, since this module uses a toolchain that is compatible with the toolchain used by the notebook diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jupyter/jupyter_paragraph_2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jupyter/jupyter_paragraph_2_metadata.json new file mode 100644 index 00000000000..0eceb68b246 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jupyter/jupyter_paragraph_2_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "jupyter", + "subtitle": "Using-extra-Python-packages", + "source_file": "../../mkdocs/docs/HPC/jupyter.md", + "title_depth": 3, + "directory": "jupyter", + "links": { + "0": "https://docs.hpc.ugent.be/troubleshooting/#module-conflicts" + }, + "parent_title": "", + "previous_title": "jupyter_paragraph_1", + "next_title": "jupyter_paragraph_3", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/jupyter/#using-extra-python-packages" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jupyter/jupyter_paragraph_3.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jupyter/jupyter_paragraph_3.txt new file mode 100644 index 00000000000..e6e9b380628 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jupyter/jupyter_paragraph_3.txt @@ -0,0 +1,6 @@ +If we use a different SciPy module that uses an incompatible toolchain, we will get a module load conflict when trying to load it (For more info on these errors, see here). +$ module load JupyterNotebook/7.2.0-GCCcore-13.2.0 +$ module load SciPy-bundle/2023.07-gfbf-2023a +Lmod has detected the following error: ... +Now that we found the right module for the notebook, add module load in the Custom code field when creating a notebook and you can make use of the packages within that notebook. + diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jupyter/jupyter_paragraph_3_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jupyter/jupyter_paragraph_3_metadata.json new file mode 100644 index 00000000000..ac4a75dfa6d --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/jupyter/jupyter_paragraph_3_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "jupyter", + "subtitle": "Using-extra-Python-packages", + "source_file": "../../mkdocs/docs/HPC/jupyter.md", + "title_depth": 3, + "directory": "jupyter", + "links": { + "0": "https://docs.hpc.ugent.be/troubleshooting/#module-conflicts" + }, + "parent_title": "", + "previous_title": "jupyter_paragraph_2", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/jupyter/#using-extra-python-packages" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/known_issues/known_issues_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/known_issues/known_issues_paragraph_1.txt new file mode 100644 index 00000000000..dedac900700 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/known_issues/known_issues_paragraph_1.txt @@ -0,0 +1,32 @@ +Known issues +This page provides details on a couple of known problems, and the workarounds that are available for them. +If you have any questions related to these issues, please contact the HPC-UGent team. +* Operation not permitted error for MPI applications +--- +Operation not permitted error for MPI applications {: #openmpi_libfabric_operation_not_permitted :} +When running an MPI application that was installed with a foss toolchain, you may run into crash with an error message like: +Failed to modify UD QP to INIT on mlx5_0: Operation not permitted +This error means that an internal problem has occurred in OpenMPI. +Cause of the problem +This problem was introduced with the OS updates that were installed on the HPC-UGent and VSC Tier-1 Hortense clusters +mid February 2024, most likely due to updating the Mellanox OFED kernel module. +It seems that having OpenMPI consider both UCX and libfabric as "backends" to use the high-speed interconnect +(InfiniBand) is causing this problem: the error message is reported by UCX, but the problem only occurs when OpenMPI +is configured to also consider libfabric. +Affected software +We have been notified that this error may occur with various applications, including (but not limited to) +CP2K, LAMMPS, netcdf4-python, SKIRT, ... +Workarounds +Use latest vsc-mympirun {: #openmpi_libfabric_mympirun :} +A workaround as been implemented in mympirun (version 5.4.0). +Make sure you use the latest version of vsc-mympirun by using the following (version-less) module load +statement in your job scripts: +module load vsc-mympirun +and launch your MPI application using the mympirun command. +For more information, see the mympirun documentation. +Configure OpenMPI to not use libfabric via environment variables {: #openmpi_libfabric_env_vars :} +If using mympirun is not an option, you can configure OpenMPI to not consider libfabric (and only use UCX) +by setting the following environment variables (in your job script or session environment): +export OMPI_MCA_btl='^uct,ofi' +export OMPI_MCA_pml='ucx' +export OMPI_MCA_mtl='^ofi' diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/known_issues/known_issues_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/known_issues/known_issues_paragraph_1_metadata.json new file mode 100644 index 00000000000..07bddef4e15 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/known_issues/known_issues_paragraph_1_metadata.json @@ -0,0 +1,17 @@ +{ + "main_title": "known_issues", + "subtitle": "Configure-OpenMPI-to-not-use-libfabric-via-environment-variables", + "source_file": "../../mkdocs/docs/HPC/known_issues.md", + "title_depth": 4, + "directory": "known_issues", + "links": { + "0": "https://www.ugent.be/hpc/en/support", + "1": "https://docs.hpc.ugent.be/known_issues/#openmpi_libfabric_operation_not_permitted", + "2": "https://docs.hpc.ugent.be/mympirun" + }, + "parent_title": "", + "previous_title": null, + "next_title": "known_issues_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/known_issues/#configure-openmpi-to-not-use-libfabric-via-environment-variables" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/known_issues/known_issues_paragraph_2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/known_issues/known_issues_paragraph_2.txt new file mode 100644 index 00000000000..13a916aec66 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/known_issues/known_issues_paragraph_2.txt @@ -0,0 +1,4 @@ +Resolution +We will re-install the affected OpenMPI installations during the scheduled maintenance of 13-17 May 2024 +(see also VSC status page). +--- \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/known_issues/known_issues_paragraph_2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/known_issues/known_issues_paragraph_2_metadata.json new file mode 100644 index 00000000000..de0499ed35f --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/known_issues/known_issues_paragraph_2_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "known_issues", + "subtitle": "Resolution", + "source_file": "../../mkdocs/docs/HPC/known_issues.md", + "title_depth": 3, + "directory": "known_issues", + "links": { + "0": "https://status.vscentrum.be" + }, + "parent_title": "", + "previous_title": "known_issues_paragraph_1", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/known_issues/#resolution" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_1.txt new file mode 100644 index 00000000000..a231341f83b --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_1.txt @@ -0,0 +1,29 @@ +Multi core jobs/Parallel Computing +Why Parallel Programming? +There are two important motivations to engage in parallel programming. +1. Firstly, the need to decrease the time to solution: distributing + your code over C cores holds the promise of speeding up execution + times by a factor C. All modern computers (and probably even your + smartphone) are equipped with multi-core processors capable of + parallel processing. +2. The second reason is problem size: distributing your code over N + nodes increases the available memory by a factor N, and thus holds + the promise of being able to tackle problems which are N times + bigger. +On a desktop computer, this enables a user to run multiple programs and +the operating system simultaneously. For scientific computing, this +means you have the ability in principle of splitting up your +computations into groups and running each group on its own core. +There are multiple different ways to achieve parallel programming. The +table below gives a (non-exhaustive) overview of problem independent +approaches to parallel programming. In addition there are many problem +specific libraries that incorporate parallel capabilities. The next +three sections explore some common approaches: (raw) threads, OpenMP and +MPI. +| Tool | Available languages binding | Limitations | +|------------------------------------------------------------------------|------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Raw threads (pthreads, boost::threading, ...) | Threading libraries are available for all common programming languages | Threading libraries are available for all common programming languages & Threads are limited to shared memory systems. They are more often used on single node systems rather than for HPC. Thread management is hard. | +| OpenMP | Fortran/C/C++ | Limited to shared memory systems, but large shared memory systems for HPC are not uncommon (e.g., SGI UV). Loops and task can be parallelized by simple insertion of compiler directives. Under the hood threads are used. Hybrid approaches exist which use OpenMP to parallelize the work load on each node and MPI (see below) for communication between nodes. | +| Lightweight threads with clever scheduling, Intel TBB, Intel Cilk Plus | C/C++ | Limited to shared memory systems, but may be combined with MPI. Thread management is taken care of by a very clever scheduler enabling the programmer to focus on parallelization itself. Hybrid approaches exist which use TBB and/or Cilk Plus to parallelise the work load on each node and MPI (see below) for communication between nodes. | +| MPI | Fortran/C/C++, Python | Applies to both distributed and shared memory systems. Cooperation between different nodes or cores is managed by explicit calls to library routines handling communication routines. | +| Global Arrays library | C/C++, Python | Mimics a global address space on distributed memory systems, by distributing arrays over many nodes and one sided communication. This library is used a lot for chemical structure calculation codes and was used in one of the first applications that broke the PetaFlop barrier. | diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_10.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_10.txt new file mode 100644 index 00000000000..97f5acea981 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_10.txt @@ -0,0 +1,19 @@ +MPI uses the notion of process rather than processor. Program copies are +mapped to processors by the MPI runtime. In that sense, the parallel +machine can map to 1 physical processor, or N where N is the total +number of processors available, or something in between. For maximum +parallel speedup, more physical processors are used. This example +adjusts its behaviour to the size of the world N, so it also seeks to +scale to the runtime configuration without compilation for each size +variation, although runtime decisions might vary depending on that +absolute amount of concurrency available. + tip + mpirun does not always do the optimal core pinning and requires a few + extra arguments to be the most efficient possible on a given system. At + Ghent we have a wrapper around mpirun called mympirun. See for more + information. + You will generally just start an MPI program on the by using mympirun + instead of + mpirun -n <--other settings> <--other optimisations> + tip + If you plan engaging in parallel programming using MPI, this book may prove useful: Parallel Programming with MPI. Peter Pacheo. Morgan Kaufmann. 1996. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_10_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_10_metadata.json new file mode 100644 index 00000000000..978fcc11b8c --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_10_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "multi_core_jobs", + "subtitle": "Parallel-Computing-with-MPI", + "source_file": "../../mkdocs/docs/HPC/multi_core_jobs.md", + "title_depth": 2, + "directory": "multi_core_jobs", + "parent_title": "", + "previous_title": "multi_core_jobs_paragraph_9", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/multi_core_jobs/#parallel-computing-with-mpi" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_1_metadata.json new file mode 100644 index 00000000000..cc1364654b7 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_1_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "multi_core_jobs", + "subtitle": "Why-Parallel-Programming", + "source_file": "../../mkdocs/docs/HPC/multi_core_jobs.md", + "title_depth": 2, + "directory": "multi_core_jobs", + "parent_title": "", + "previous_title": null, + "next_title": "multi_core_jobs_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/multi_core_jobs/#why-parallel-programming" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_2.txt new file mode 100644 index 00000000000..974014a9e0c --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_2.txt @@ -0,0 +1,75 @@ + tip + You can request more nodes/cores by adding following line to your run script. + #PBS -l nodes=2:ppn=10 + This queues a job that claims 2 nodes and 10 cores. + warning + Just requesting more nodes and/or cores does not mean that your job will automatically run faster. + You can find more about this here. +Parallel Computing with threads +Multi-threading is a widespread programming and execution model that +allows multiple threads to exist within the context of a single process. +These threads share the process' resources, but are able to execute +independently. The threaded programming model provides developers with a +useful abstraction of concurrent execution. Multi-threading can also be +applied to a single process to enable parallel execution on a +multiprocessing system. +This advantage of a multithreaded program allows it to operate faster on +computer systems that have multiple CPUs or across a cluster of machines +--- because the threads of the program naturally lend themselves to +truly concurrent execution. In such a case, the programmer needs to be +careful to avoid race conditions, and other non-intuitive behaviours. In +order for data to be correctly manipulated, threads will often need to +synchronise in time in order to process the data in the correct order. +Threads may also require mutually exclusive operations (often +implemented using semaphores) in order to prevent common data from being +simultaneously modified, or read while in the process of being modified. +Careless use of such primitives can lead to deadlocks. +Threads are a way that a program can spawn concurrent units of +processing that can then be delegated by the operating system to +multiple processing cores. Clearly the advantage of a multithreaded +program (one that uses multiple threads that are assigned to multiple +processing cores) is that you can achieve big speedups, as all cores of +your CPU (and all CPUs if you have more than one) are used at the same +time. +Here is a simple example program that spawns 5 threads, where each one +runs a simple function that only prints "Hello from thread". +Go to the example directory: +cd ~/examples/Multi-core-jobs-Parallel-Computing + note + If the example directory is not yet present, copy it to your home directory: + cp -r /apps/gent/tutorials/Intro-HPC/examples ~/ +Study the example first: +/* + * VSC : Flemish Supercomputing Centre + * Tutorial : Introduction to HPC + * Description: Showcase of working with threads + */ +#include +#include +#include +#define NTHREADS 5 +void *myFun(void *x) +{ + int tid; + tid = *((int *) x); + printf("Hello from thread %d!\n", tid); + return NULL; +} +int main(int argc, char *argv[]) +{ + pthread_t threads[NTHREADS]; + int thread_args[NTHREADS]; + int rc, i; + /* spawn the threads */ + for (i=0; i +main () { +int var1, var2, var3; +// Serial code +// Beginning of parallel section. Fork a team of threads. +// Specify variable scoping +#pragma omp parallel private(var1, var2) shared(var3) + { + // Parallel section executed by all threads + // All threads join master thread and disband + } +// Resume serial code +} diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_3_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_3_metadata.json new file mode 100644 index 00000000000..6275b4692ba --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_3_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "multi_core_jobs", + "subtitle": "Parallel-Computing-with-OpenMP", + "source_file": "../../mkdocs/docs/HPC/multi_core_jobs.md", + "title_depth": 2, + "directory": "multi_core_jobs", + "parent_title": "", + "previous_title": "multi_core_jobs_paragraph_2", + "next_title": "multi_core_jobs_paragraph_4", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/multi_core_jobs/#parallel-computing-with-openmp" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_4.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_4.txt new file mode 100644 index 00000000000..6d83a39c499 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_4.txt @@ -0,0 +1,59 @@ +Private versus Shared variables +By using the private() and shared() clauses, you can specify variables +within the parallel region as being shared, i.e., visible and accessible by +all threads simultaneously, or private, i.e., private to each thread, meaning +each thread will have its own local copy. In the code example below for +parallelising a for loop, you can see that we specify the thread_id and +nloops variables as private. +Parallelising for loops with OpenMP +Parallelising for loops is really simple (see code below). By default, +loop iteration counters in OpenMP loop constructs (in this case the i +variable) in the for loop are set to private variables. +/* + * VSC : Flemish Supercomputing Centre + * Tutorial : Introduction to HPC + * Description: Showcase program for OMP loops + */ +/* OpenMP_loop.c */ +#include +#include +int main(int argc, char **argv) +{ + int i, thread_id, nloops; +#pragma omp parallel private(thread_id, nloops) + { + nloops = 0; +#pragma omp for + for (i=0; i<1000; ++i) + { + ++nloops; + } + thread_id = omp_get_thread_num(); + printf("Thread %d performed %d iterations of the loop.\n", thread_id, nloops ); + } + return 0; +} +And compile it (whilst including the "openmp" library) and run and +test it on the login-node: +$ module load GCC +$ gcc -fopenmp -o omp1 omp1.c +$ ./omp1 +Thread 6 performed 125 iterations of the loop. +Thread 7 performed 125 iterations of the loop. +Thread 5 performed 125 iterations of the loop. +Thread 4 performed 125 iterations of the loop. +Thread 0 performed 125 iterations of the loop. +Thread 2 performed 125 iterations of the loop. +Thread 3 performed 125 iterations of the loop. +Thread 1 performed 125 iterations of the loop. +Now run it in the cluster and check the result again. +$ qsub omp1.pbs +$ cat omp1.pbs.o* +Thread 1 performed 125 iterations of the loop. +Thread 4 performed 125 iterations of the loop. +Thread 3 performed 125 iterations of the loop. +Thread 0 performed 125 iterations of the loop. +Thread 5 performed 125 iterations of the loop. +Thread 7 performed 125 iterations of the loop. +Thread 2 performed 125 iterations of the loop. +Thread 6 performed 125 iterations of the loop. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_4_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_4_metadata.json new file mode 100644 index 00000000000..e10a4b55313 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_4_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "multi_core_jobs", + "subtitle": "Parallelising-for-loops-with-OpenMP", + "source_file": "../../mkdocs/docs/HPC/multi_core_jobs.md", + "title_depth": 3, + "directory": "multi_core_jobs", + "parent_title": "", + "previous_title": "multi_core_jobs_paragraph_3", + "next_title": "multi_core_jobs_paragraph_5", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/multi_core_jobs/#parallelising-for-loops-with-openmp" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_5.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_5.txt new file mode 100644 index 00000000000..cb5eab6204b --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_5.txt @@ -0,0 +1,55 @@ +Critical Code +Using OpenMP you can specify something called a "critical" section of +code. This is code that is performed by all threads, but is only +performed one thread at a time (i.e., in serial). This provides a convenient way of letting +you do things like updating a global variable with local results from +each thread, and you don't have to worry about things like other threads +writing to that global variable at the same time (a collision). +/* + * VSC : Flemish Supercomputing Centre + * Tutorial : Introduction to HPC + * Description: OpenMP Test Program + */ +#include +#include +int main(int argc, char *argv[]) +{ + int i, thread_id; + int glob_nloops, priv_nloops; + glob_nloops = 0; + // parallelize this chunk of code + #pragma omp parallel private(priv_nloops, thread_id) + { + priv_nloops = 0; + thread_id = omp_get_thread_num(); + // parallelize this for loop + #pragma omp for + for (i=0; i<100000; ++i) + { + ++priv_nloops; + } + // make this a "critical" code section + #pragma omp critical + { + printf("Thread %d is adding its iterations (%d) to sum (%d), ", thread_id, priv_nloops, glob_nloops); + glob_nloops += priv_nloops; + printf("total is now %d.\n", glob_nloops); + } + } + printf("Total # loop iterations is %d\n", glob_nloops); + return 0; +} +And compile it (whilst including the "openmp" library) and run and +test it on the login-node: +$ module load GCC +$ gcc -fopenmp -o omp2 omp2.c +$ ./omp2 +Thread 3 is adding its iterations (12500) to sum (0), total is now 12500. +Thread 7 is adding its iterations (12500) to sum (12500), total is now 25000. +Thread 5 is adding its iterations (12500) to sum (25000), total is now 37500. +Thread 6 is adding its iterations (12500) to sum (37500), total is now 50000. +Thread 2 is adding its iterations (12500) to sum (50000), total is now 62500. +Thread 4 is adding its iterations (12500) to sum (62500), total is now 75000. +Thread 1 is adding its iterations (12500) to sum (75000), total is now 87500. +Thread 0 is adding its iterations (12500) to sum (87500), total is now 100000. +Total # loop iterations is 100000 diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_5_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_5_metadata.json new file mode 100644 index 00000000000..0c6521fc05a --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_5_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "multi_core_jobs", + "subtitle": "Critical-Code", + "source_file": "../../mkdocs/docs/HPC/multi_core_jobs.md", + "title_depth": 3, + "directory": "multi_core_jobs", + "parent_title": "", + "previous_title": "multi_core_jobs_paragraph_4", + "next_title": "multi_core_jobs_paragraph_6", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/multi_core_jobs/#critical-code" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_6.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_6.txt new file mode 100644 index 00000000000..ac338a5994e --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_6.txt @@ -0,0 +1,48 @@ +Now run it in the cluster and check the result again. +$ qsub omp2.pbs +$ cat omp2.pbs.o* +Thread 2 is adding its iterations (12500) to sum (0), total is now 12500. +Thread 0 is adding its iterations (12500) to sum (12500), total is now 25000. +Thread 1 is adding its iterations (12500) to sum (25000), total is now 37500. +Thread 4 is adding its iterations (12500) to sum (37500), total is now 50000. +Thread 7 is adding its iterations (12500) to sum (50000), total is now 62500. +Thread 3 is adding its iterations (12500) to sum (62500), total is now 75000. +Thread 5 is adding its iterations (12500) to sum (75000), total is now 87500. +Thread 6 is adding its iterations (12500) to sum (87500), total is now 100000. +Total # loop iterations is 100000 +Reduction +Reduction refers to the process of combining the results of several +sub-calculations into a final result. This is a very common paradigm +(and indeed the so-called "map-reduce" framework used by Google and +others is very popular). Indeed we used this paradigm in the code +example above, where we used the "critical code" directive to accomplish +this. The map-reduce paradigm is so common that OpenMP has a specific +directive that allows you to more easily implement this. +/* + * VSC : Flemish Supercomputing Centre + * Tutorial : Introduction to HPC + * Description: OpenMP Test Program + */ +#include +#include +int main(int argc, char *argv[]) +{ + int i, thread_id; + int glob_nloops, priv_nloops; + glob_nloops = 0; + // parallelize this chunk of code + #pragma omp parallel private(priv_nloops, thread_id) reduction(+:glob_nloops) + { + priv_nloops = 0; + thread_id = omp_get_thread_num(); + // parallelize this for loop + #pragma omp for + for (i=0; i<100000; ++i) + { + ++priv_nloops; + } + glob_nloops += priv_nloops; + } + printf("Total # loop iterations is %d\n", glob_nloops); + return 0; +} diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_6_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_6_metadata.json new file mode 100644 index 00000000000..facf7808e12 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_6_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "multi_core_jobs", + "subtitle": "Reduction", + "source_file": "../../mkdocs/docs/HPC/multi_core_jobs.md", + "title_depth": 3, + "directory": "multi_core_jobs", + "parent_title": "", + "previous_title": "multi_core_jobs_paragraph_5", + "next_title": "multi_core_jobs_paragraph_7", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/multi_core_jobs/#reduction" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_7.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_7.txt new file mode 100644 index 00000000000..78b2e7e51bb --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_7.txt @@ -0,0 +1,48 @@ +And compile it (whilst including the "openmp" library) and run and +test it on the login-node: +$ module load GCC +$ gcc -fopenmp -o omp3 omp3.c +$ ./omp3 +Total # loop iterations is 100000 +Now run it in the cluster and check the result again. +$ qsub omp3.pbs +$ cat omp3.pbs.o* +Total # loop iterations is 100000 +Other OpenMP directives +There are a host of other directives you can issue using OpenMP. +Some other clauses of interest are: +1. barrier: each thread will wait until all threads have reached this + point in the code, before proceeding +2. nowait: threads will not wait until everybody is finished +3. schedule(type, chunk) allows you to specify how tasks are spawned + out to threads in a for loop. There are three types of scheduling + you can specify +4. if: allows you to parallelise only if a certain condition is met +5. ... and a host of others + tip + If you plan engaging in parallel programming using OpenMP, this book may + prove useful: *Using OpenMP - Portable Shared Memory Parallel + Programming*. By Barbara Chapman Gabriele Jost and Ruud van der Pas + Scientific and Engineering Computation. 2005. +Parallel Computing with MPI +The Message Passing Interface (MPI) is a standard defining core syntax +and semantics of library routines that can be used to implement parallel +programming in C (and in other languages as well). There are several +implementations of MPI such as Open MPI, Intel MPI, M(VA)PICH and +LAM/MPI. +In the context of this tutorial, you can think of MPI, in terms of its +complexity, scope and control, as sitting in between programming with +Pthreads, and using a high-level API such as OpenMP. For a Message +Passing Interface (MPI) application, a parallel task usually consists of +a single executable running concurrently on multiple processors, with +communication between the processes. This is shown in the following +diagram: +The process numbers 0, 1 and 2 represent the process rank and have +greater or less significance depending on the processing paradigm. At +the minimum, Process 0 handles the input/output and determines what +other processes are running. +The MPI interface allows you to manage allocation, communication, and +synchronisation of a set of processes that are mapped onto multiple +nodes, where each node can be a core within a single CPU, or CPUs within +a single machine, or even across multiple machines (as long as they are +networked together). diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_7_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_7_metadata.json new file mode 100644 index 00000000000..2c9a51cccaa --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_7_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "multi_core_jobs", + "subtitle": "Parallel-Computing-with-MPI", + "source_file": "../../mkdocs/docs/HPC/multi_core_jobs.md", + "title_depth": 2, + "directory": "multi_core_jobs", + "parent_title": "", + "previous_title": "multi_core_jobs_paragraph_6", + "next_title": "multi_core_jobs_paragraph_8", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/multi_core_jobs/#parallel-computing-with-mpi" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_8.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_8.txt new file mode 100644 index 00000000000..d577fc208a6 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_core_jobs/multi_core_jobs_paragraph_8.txt @@ -0,0 +1,71 @@ +One context where MPI shines in particular is the ability to easily take +advantage not just of multiple cores on a single machine, but to run +programs on clusters of several machines. Even if you don't have a +dedicated cluster, you could still write a program using MPI that could +run your program in parallel, across any collection of computers, as +long as they are networked together. +Here is a "Hello World" program in MPI written in C. In this example, we +send a "Hello" message to each processor, manipulate it trivially, +return the results to the main process, and print the messages. +Study the MPI-programme and the PBS-file: +/* + * VSC : Flemish Supercomputing Centre + * Tutorial : Introduction to HPC + * Description: "Hello World" MPI Test Program + */ +#include +#include + #include + #include + #include + + #define BUFSIZE 128 + #define TAG 0 + + int main(int argc, char *argv[]) + { + char idstr[32]; + char buff[BUFSIZE]; + int numprocs; + int myid; + int i; + MPI_Status stat; + /* MPI programs start with MPI_Init; all 'N' processes exist thereafter */ + MPI_Init(&argc,&argv); + /* find out how big the SPMD world is */ + MPI_Comm_size(MPI_COMM_WORLD,&numprocs); + /* and this processes' rank is */ + MPI_Comm_rank(MPI_COMM_WORLD,&myid); + + /* At this point, all programs are running equivalently, the rank + distinguishes the roles of the programs in the SPMD model, with + rank 0 often used specially... */ + if(myid == 0) + { + printf("%d: We have %d processors\n", myid, numprocs); + for(i=1;i ./output/$5 +echo "Calculations done, no results" >> ./output/$5 +Using the "worker framework", a feature akin to job arrays can be used +with minimal modifications to the job script: +#!/bin/bash -l +#PBS -l nodes=1:ppn=8 +#PBS -l walltime=04:00:00 +cd $PBS_O_WORKDIR +INPUT_FILE="input_${PBS_ARRAYID}.dat" +OUTPUT_FILE="output_${PBS_ARRAYID}.dat" +./test_set ${PBS_ARRAYID} -input ${INPUT_FILE} -output ${OUTPUT_FILE} +Note that +1. the number of CPUs is increased to 8 (ppn=1 is replaced by ppn=8); + and +2. the walltime has been modified (walltime=00:15:00 is replaced by + walltime=04:00:00). +The job is now submitted as follows: +$ module load worker/1.6.12-foss-2021b +$ wsub -t 1-100 -batch test_set.pbs +total number of work items: 100 +123456 diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_5_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_5_metadata.json new file mode 100644 index 00000000000..c2f9422ca31 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_5_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "multi_job_submission", + "subtitle": "The-Worker-framework-Job-arrays", + "source_file": "../../mkdocs/docs/HPC/multi_job_submission.md", + "title_depth": 2, + "directory": "multi_job_submission", + "parent_title": "", + "previous_title": "multi_job_submission_paragraph_4", + "next_title": "multi_job_submission_paragraph_6", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/multi_job_submission/#the-worker-framework-job-arrays" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_6.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_6.txt new file mode 100644 index 00000000000..190d048e407 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_6.txt @@ -0,0 +1,49 @@ +The "test_set" program will now be run for all 100 input files -- 8 +concurrently -- until all computations are done. Again, a computation +for an individual input file, or, equivalently, an array id, is called a +work item in Worker speak. +Note that in contrast to TORQUE job arrays, a worker job array only +submits a single job. +$ qstat +Job id Name User Time Use S Queue +--------------- ------------- --------- ---- ----- - ----- +123456 test_set.pbs vsc40000 0 Q +And you can now check the generated output files: +$ more ./output/output_99.dat +This is output file #99 +Calculations done, no results +MapReduce: prologues and epilogue +Often, an embarrassingly parallel computation can be abstracted to three +simple steps: +1. a preparation phase in which the data is split up into smaller, more + manageable chunks; +2. on these chunks, the same algorithm is applied independently (these + are the work items); and +3. the results of the computations on those chunks are aggregated into, + e.g., a statistical description of some sort. +The Worker framework directly supports this scenario by using a prologue +(pre-processing) and an epilogue (post-processing). The former is +executed just once before work is started on the work items, the latter +is executed just once after the work on all work items has finished. +Technically, the master, i.e., the process that is responsible for +dispatching work and logging progress, executes the prologue and +epilogue. +cd ~/examples/Multi-job-submission/map_reduce +The script "pre.sh" prepares the data by creating 100 different +input-files, and the script "post.sh" aggregates (concatenates) the +data. +First study the scripts: +#!/bin/bash +# Check if the input Directory exists +if [ ! -d "./input" ] ; then + mkdir ./input +fi +# Just generate all dummy input files +for i in {1..100}; +do + echo "This is input file #$i" > ./input/input_$i.dat + echo "Parameter #1 = $i" >> ./input/input_$i.dat + echo "Parameter #2 = 25.67" >> ./input/input_$i.dat + echo "Parameter #3 = Batch" >> ./input/input_$i.dat + echo "Parameter #4 = 0x562867" >> ./input/input_$i.dat +done diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_6_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_6_metadata.json new file mode 100644 index 00000000000..a370264b785 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_6_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "multi_job_submission", + "subtitle": "MapReduce-prologues-and-epilogue", + "source_file": "../../mkdocs/docs/HPC/multi_job_submission.md", + "title_depth": 2, + "directory": "multi_job_submission", + "parent_title": "", + "previous_title": "multi_job_submission_paragraph_5", + "next_title": "multi_job_submission_paragraph_7", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/multi_job_submission/#mapreduce-prologues-and-epilogue" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_7.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_7.txt new file mode 100644 index 00000000000..202538ee278 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_7.txt @@ -0,0 +1,59 @@ +#!/bin/bash +# Check if the input Directory exists +if [ ! -d "./output" ] ; then + echo "The output directory does not exist!" + exit +fi +# Just concatenate all output files +touch all_output.txt +for i in {1..100}; +do + cat ./output/output_$i.dat >> all_output.txt +done +Then one can submit a MapReduce style job as follows: +$ wsub -prolog pre.sh -batch test_set.pbs -epilog post.sh -t 1-100 +total number of work items: 100 +123456 +$ cat all_output.txt +... +$ rm -r -f ./output/ +Note that the time taken for executing the prologue and the epilogue +should be added to the job's total walltime. +Some more on the Worker Framework +Using Worker efficiently +The "Worker Framework" is implemented using MPI, so it is not restricted +to a single compute nodes, it scales well to multiple nodes. However, +remember that jobs requesting a large number of nodes typically spend +quite some time in the queue. +The "Worker Framework" will be effective when +1. work items, i.e., individual computations, are neither too short, + nor too long (i.e., from a few minutes to a few hours); and, +2. when the number of work items is larger than the number of CPUs + involved in the job (e.g., more than 30 for 8 CPUs). +Monitoring a worker job +Since a Worker job will typically run for several hours, it may be +reassuring to monitor its progress. Worker keeps a log of its activity +in the directory where the job was submitted. The log's name is derived +from the job's name and the job's ID, i.e., it has the form +.log. For the running example, this could be +run.pbs.log123456, assuming the job's ID is 123456. To keep an eye on the +progress, one can use: +tail -f run.pbs.log123456 +Alternatively, wsummarize, a Worker command that summarises a log +file, can be used: +watch -n 60 wsummarize run.pbs.log123456 +This will summarise the log file every 60 seconds. +Time limits for work items +Sometimes, the execution of a work item takes longer than expected, or +worse, some work items get stuck in an infinite loop. This situation is +unfortunate, since it implies that work items that could successfully +execute are not even started. Again, the Worker framework offers a +simple and yet versatile solution. If we want to limit the execution of +each work item to at most 20 minutes, this can be accomplished by +modifying the script of the running example. +#!/bin/bash -l +#PBS -l nodes=1:ppn=8 +#PBS -l walltime=04:00:00 +module load timedrun/1.0 +cd $PBS_O_WORKDIR +timedrun -t 00:20:00 weather -t $temperature -p $pressure -v $volume diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_7_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_7_metadata.json new file mode 100644 index 00000000000..07d9c4a1015 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_7_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "multi_job_submission", + "subtitle": "Time-limits-for-work-items", + "source_file": "../../mkdocs/docs/HPC/multi_job_submission.md", + "title_depth": 3, + "directory": "multi_job_submission", + "parent_title": "", + "previous_title": "multi_job_submission_paragraph_6", + "next_title": "multi_job_submission_paragraph_8", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/multi_job_submission/#time-limits-for-work-items" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_8.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_8.txt new file mode 100644 index 00000000000..d6d30f7a58e --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_8.txt @@ -0,0 +1,63 @@ +Note that it is trivial to set individual time constraints for work +items by introducing a parameter, and including the values of the latter +in the CSV file, along with those for the temperature, pressure and +volume. +Also note that "timedrun" is in fact offered in a module of its own, so +it can be used outside the Worker framework as well. +Resuming a Worker job +Unfortunately, walltime is sometimes underestimated. When using the +Worker framework, this implies that not all work items will have been +processed. Worker makes it very easy to resume such a job without having +to figure out which work items did complete successfully, and which +remain to be computed. Suppose the job that did not complete all its +work items had ID "445948". +wresume -jobid 123456 +This will submit a new job that will start to work on the work items +that were not done yet. Note that it is possible to change almost all +job parameters when resuming, specifically the requested resources such +as the number of cores and the walltime. +wresume -l walltime=1:30:00 -jobid 123456 +Work items may fail to complete successfully for a variety of reasons, +e.g., a data file that is missing, a (minor) programming error, etc. +Upon resuming a job, the work items that failed are considered to be +done, so resuming a job will only execute work items that did not +terminate either successfully, or reporting a failure. It is also +possible to retry work items that failed (preferably after the glitch +why they failed was fixed). +wresume -jobid 123456 -retry +By default, a job's prologue is not executed when it is resumed, while +its epilogue is. "wresume" has options to modify this default behaviour. +Further information +This how-to introduces only Worker's basic features. The wsub command +has some usage information that is printed when the -help option is +specified: +$ wsub -help +### usage: wsub -batch <batch-file> +# [-data <data-files>] +# [-prolog <prolog-file>] +# [-epilog <epilog-file>] +# [-log <log-file>] +# [-mpiverbose] +# [-dryrun] [-verbose] +# [-quiet] [-help] +# [-t <array-req>] +# [<pbs-qsub-options>] +# +# -batch <batch-file> : batch file template, containing variables to be +# replaced with data from the data file(s) or the +# PBS array request option +# -data <data-files> : comma-separated list of data files (default CSV +# files) used to provide the data for the work +# items +# -prolog <prolog-file> : prolog script to be executed before any of the +# work items are executed +# -epilog <epilog-file> : epilog script to be executed after all the work +# items are executed +# -mpiverbose : pass verbose flag to the underlying MPI program +# -verbose : feedback information is written to standard error +# -dryrun : run without actually submitting the job, useful +# -quiet : don't show information +# -help : print this help message +# -t <array-req> : qsub's PBS array request options, e.g., 1-10 +# <pbs-qsub-options> : options passed on to the queue submission +# command diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_8_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_8_metadata.json new file mode 100644 index 00000000000..39b01022c56 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_8_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "multi_job_submission", + "subtitle": "Further-information", + "source_file": "../../mkdocs/docs/HPC/multi_job_submission.md", + "title_depth": 3, + "directory": "multi_job_submission", + "parent_title": "", + "previous_title": "multi_job_submission_paragraph_7", + "next_title": "multi_job_submission_paragraph_9", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/multi_job_submission/#further-information" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_9.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_9.txt new file mode 100644 index 00000000000..e72a306429e --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_9.txt @@ -0,0 +1,12 @@ +Troubleshooting +Error: An ORTE daemon has unexpectedly failed after launch and before communicating back to mpirun +When submitting a Worker job, you might encounter the following error: +An ORTE daemon has unexpectedly failed after launch and before communicating back to mpirun. +This error can occur when the foss toolchain version of worker is loaded. Instead, try loading an iimpi toolchain version of worker. +to check for the available versions of worker, use the following command: +$ module avail worker +[^1]: MapReduce: 'Map' refers to the map pattern in which every item in + a collection is mapped onto a new value by applying a given + function, while "reduce" refers to the reduction pattern which + condenses or reduces a collection of previously computed results to + a single value. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_9_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_9_metadata.json new file mode 100644 index 00000000000..603f054702f --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/multi_job_submission/multi_job_submission_paragraph_9_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "multi_job_submission", + "subtitle": "Error-An-ORTE-daemon-has-unexpectedly-failed-after-launch-and-before-communicating-back-to-mpirun", + "source_file": "../../mkdocs/docs/HPC/multi_job_submission.md", + "title_depth": 3, + "directory": "multi_job_submission", + "parent_title": "", + "previous_title": "multi_job_submission_paragraph_8", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/multi_job_submission/#error-an-orte-daemon-has-unexpectedly-failed-after-launch-and-before-communicating-back-to-mpirun" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/mympirun/mympirun_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/mympirun/mympirun_paragraph_1.txt new file mode 100644 index 00000000000..d8bda282f6b --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/mympirun/mympirun_paragraph_1.txt @@ -0,0 +1,40 @@ +Mympirun +mympirun is a tool to make it easier for users of HPC clusters to run +MPI programs with good performance. We strongly recommend to use +mympirun instead of impirun. +In this chapter, we give a high-level overview. For a more detailed +description of all options, see the [vsc-mympirun +README](https://github.com/hpcugent/vsc-mympirun/blob/master/README.md). +Basic usage +Before using mympirun, we first need to load its module: +module load vsc-mympirun +As an exception, we don't specify a version here. The reason is that we +want to ensure that the latest version of the mympirun script is +always used, since it may include important bug fixes or improvements. +The most basic form of using mympirun is +mympirun [mympirun options] your_program [your_program options]. +For example, to run a program named example and give it a single +argument (5), we can run it with mympirun example 5. +Controlling number of processes +There are four options you can choose from to control the number of +processes mympirun will start. In the following example, the program +mpi_hello prints a single line: +Hello world from processor ... (the sourcecode of mpi_hello +is [available in the vsc-mympirun +repository](https://github.com/hpcugent/vsc-mympirun/blob/master/testscripts/mpi_helloworld.c)). +By default, mympirun starts one process per core on every node you +assigned. So if you assigned 2 nodes with 16 cores each, mympirun will +start 2 . 16 = 32 test processes in total. +--hybrid/-h +This is the most commonly used option for controlling the number of +processing. +The --hybrid option requires a positive number. This number specifies +the number of processes started on each available physical node. It +will ignore the number of available cores per node. +$ echo $PBS_NUM_NODES +2 +$ mympirun --hybrid 2 ./mpihello +Hello world from processor node3400.doduo.os, rank 1 out of 4 processors +Hello world from processor node3401.doduo.os, rank 3 out of 4 processors +Hello world from processor node3401.doduo.os, rank 2 out of 4 processors +Hello world from processor node3400.doduo.os, rank 0 out of 4 processors diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/mympirun/mympirun_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/mympirun/mympirun_paragraph_1_metadata.json new file mode 100644 index 00000000000..f0bc13f5188 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/mympirun/mympirun_paragraph_1_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "mympirun", + "subtitle": "`-hybrid``-h`", + "source_file": "../../mkdocs/docs/HPC/mympirun.md", + "title_depth": 3, + "directory": "mympirun", + "parent_title": "", + "previous_title": null, + "next_title": "mympirun_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/mympirun/#hybrid-h" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/mympirun/mympirun_paragraph_2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/mympirun/mympirun_paragraph_2.txt new file mode 100644 index 00000000000..1d2fe8f72a0 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/mympirun/mympirun_paragraph_2.txt @@ -0,0 +1,15 @@ +Other options +There's also --universe, which sets the exact amount of processes +started by mympirun; --double, which uses double the amount of +processes it normally would; and --multi that does the same as +--double, but takes a multiplier (instead of the implied factor 2 with +--double). +See [vsc-mympirun +README](https://github.com/hpcugent/vsc-mympirun/blob/master/README.md) +for a detailed explanation of these options. +Dry run +You can do a so-called "dry run", which doesn't have any side-effects, +but just prints the command that mympirun would execute. You enable +this with the --dry-run flag: +$ mympirun --dry-run ./mpi_hello +mpirun ... -genv I_MPI_FABRICS shm:dapl ... -np 16 ... ./mpi_hello diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/mympirun/mympirun_paragraph_2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/mympirun/mympirun_paragraph_2_metadata.json new file mode 100644 index 00000000000..9be5a4d18b2 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/mympirun/mympirun_paragraph_2_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "mympirun", + "subtitle": "Dry-run", + "source_file": "../../mkdocs/docs/HPC/mympirun.md", + "title_depth": 2, + "directory": "mympirun", + "parent_title": "", + "previous_title": "mympirun_paragraph_1", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/mympirun/#dry-run" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_1.txt new file mode 100644 index 00000000000..53686eb09ec --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_1.txt @@ -0,0 +1,37 @@ +OpenFOAM +In this chapter, we outline best practices for using the centrally +provided OpenFOAM installations on the VSC HPC infrastructure. +Different OpenFOAM releases +There are currently three different sets of versions of OpenFOAM +available, each with its own versioning scheme: +- OpenFOAM versions released via http://openfoam.com: v3.0+, + v1706 + - see also http://openfoam.com/history/ +- OpenFOAM versions released via https://openfoam.org: v4.1, + v5.0 + - see also https://openfoam.org/download/history/ +- OpenFOAM versions released via + http://wikki.gridcore.se/foam-extend: v3.1 +Make sure you know which flavor of OpenFOAM you want to use, since there +are important differences between the different versions w.r.t. +features. If the OpenFOAM version you need is not available yet, see I want to use software that is not available on the clusters yet. +Documentation & training material +The best practices outlined here focus specifically on the use of +OpenFOAM on the VSC HPC infrastructure. As such, they are intended to +augment the existing OpenFOAM documentation rather than replace it. For +more general information on using OpenFOAM, please refer to: +- OpenFOAM websites: + - https://openfoam.com + - https://openfoam.org + - http://wikki.gridcore.se/foam-extend +- OpenFOAM user guides: + - https://www.openfoam.com/documentation/user-guide + - https://cfd.direct/openfoam/user-guide/ +- OpenFOAM C++ source code guide: https://cpp.openfoam.org +- tutorials: https://wiki.openfoam.com/Tutorials +- recordings of "Introduction to OpenFOAM" training session at + UGent (May 2016): + +Other useful OpenFOAM documentation: +- https://github.com/ParticulateFlow/OSCCAR-doc/blob/master/openFoamUserManual_PFM.pdf +- http://www.dicat.unige.it/guerrero/openfoam.html diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_1_metadata.json new file mode 100644 index 00000000000..1c16a41507c --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "openFOAM", + "subtitle": "Documentation-&-training-material", + "source_file": "../../mkdocs/docs/HPC/openFOAM.md", + "title_depth": 2, + "directory": "openFOAM", + "links": { + "0": "https://docs.hpc.ugent.be/FAQ/#i-want-to-use-software-that-is-not-available-on-the-clusters-yet" + }, + "parent_title": "", + "previous_title": null, + "next_title": "openFOAM_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/openFOAM/#documentation--training-material" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_2.txt new file mode 100644 index 00000000000..eeab36ed281 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_2.txt @@ -0,0 +1,35 @@ +Preparing the environment +To prepare the environment of your shell session or job for using +OpenFOAM, there are a couple of things to take into account. +Picking and loading an OpenFOAM module +First of all, you need to pick and load one of the available OpenFOAM +modules. To get an overview of the available modules, run +'module avail OpenFOAM'. For example: +$ module avail OpenFOAM +------------------ /apps/gent/CO7/sandybridge/modules/all ------------------ + OpenFOAM/v1712-foss-2017b OpenFOAM/4.1-intel-2017a + OpenFOAM/v1712-intel-2017b OpenFOAM/5.0-intel-2017a + OpenFOAM/2.2.2-intel-2017a OpenFOAM/5.0-intel-2017b + OpenFOAM/2.2.2-intel-2018a OpenFOAM/5.0-20180108-foss-2018a + OpenFOAM/2.3.1-intel-2017a OpenFOAM/5.0-20180108-intel-2017b + OpenFOAM/2.4.0-intel-2017a OpenFOAM/5.0-20180108-intel-2018a + OpenFOAM/3.0.1-intel-2016b OpenFOAM/6-intel-2018a (D) + OpenFOAM/4.0-intel-2016b +To pick a module, take into account the differences between the +different OpenFOAM versions w.r.t. features and API (see also Different OpenFOAM releases). If +multiple modules are available that fulfill your requirements, give +preference to those providing a more recent OpenFOAM version, and to the +ones that were installed with a more recent compiler toolchain; for +example, prefer a module that includes intel-b in its name over one +that includes intel-a. +To prepare your environment for using OpenFOAM, load the OpenFOAM +module you have picked; for example: +module load OpenFOAM/11-foss-2023a +Sourcing the $FOAM_BASH script +OpenFOAM provides a script that you should source to further prepare +the environment. This script will define some additional environment +variables that are required to use OpenFOAM. The OpenFOAM modules +define an environment variable named FOAM_BASH that specifies the +location to this script. Assuming you are using bash in your shell +session or job script, you should always run the following command after +loading an OpenFOAM module: diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_2_metadata.json new file mode 100644 index 00000000000..a388a07220f --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_2_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "openFOAM", + "subtitle": "Sourcing-the-`$FOAM_BASH`-script", + "source_file": "../../mkdocs/docs/HPC/openFOAM.md", + "title_depth": 3, + "directory": "openFOAM", + "links": { + "0": "https://docs.hpc.ugent.be/openFOAM/#different-openfoam-releases" + }, + "parent_title": "", + "previous_title": "openFOAM_paragraph_1", + "next_title": "openFOAM_paragraph_3", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/openFOAM/#sourcing-the-foambash-script" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_3.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_3.txt new file mode 100644 index 00000000000..36d6dd284f8 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_3.txt @@ -0,0 +1,47 @@ +source $FOAM_BASH +Defining utility functions used in tutorial cases +If you would like to use the getApplication, runApplication, +runParallel, cloneCase and/or compileApplication functions that +are used in OpenFOAM tutorials, you also need to source the +RunFunctions script: +source $WM_PROJECT_DIR/bin/tools/RunFunctions +Note that this needs to be done after sourcing $FOAM_BASH to make sure +$WM_PROJECT_DIR is defined. +Dealing with floating-point errors +If you are seeing Floating Point Exception errors, you can undefine +the $FOAM_SIGFPE environment variable that is defined by the +$FOAM_BASH script as follows: +unset $FOAM_SIGFPE +Note that this only prevents OpenFOAM from propagating floating point +exceptions, which then results in terminating the simulation. However, +it does not prevent that illegal operations (like a division by zero) +are being executed; if NaN values appear in your results, floating +point errors are occurring. +As such, *you should not* use this in production runs. Instead, you should track down the root cause of the floating +point errors, and try to prevent them from occurring at all. +OpenFOAM workflow +The general workflow for OpenFOAM consists of multiple steps. Prior to +running the actual simulation, some pre-processing needs to be done: +- generate the mesh; +- decompose the domain into subdomains using decomposePar (only for + parallel OpenFOAM simulations); +After running the simulation, some post-processing steps are typically +performed: +- reassemble the decomposed domain using reconstructPar (only for + parallel OpenFOAM simulations, and optional since some + postprocessing can also be done on decomposed cases); +- evaluate or further process the simulation results, either visually + using ParaView (for example, via the paraFoam tool; use + paraFoam -builtin for decomposed cases) or using command-line + tools like postProcess; see also + https://cfd.direct/openfoam/user-guide/postprocessing. +Depending on the size of the domain and the desired format of the +results, these pre- and post-processing steps can be run either +before/after the job running the actual simulation, either on the HPC +infrastructure or elsewhere, or as a part of the job that runs the +OpenFOAM simulation itself. +Do make sure you are using the same OpenFOAM version in each of the +steps. Meshing can be done sequentially (i.e., on a single core) using +for example blockMesh, or in parallel using more advanced meshing +tools like snappyHexMesh, which is highly recommended for large cases. +For more details, see https://cfd.direct/openfoam/user-guide/mesh/. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_3_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_3_metadata.json new file mode 100644 index 00000000000..2b59f60a504 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_3_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "openFOAM", + "subtitle": "OpenFOAM-workflow", + "source_file": "../../mkdocs/docs/HPC/openFOAM.md", + "title_depth": 2, + "directory": "openFOAM", + "parent_title": "", + "previous_title": "openFOAM_paragraph_2", + "next_title": "openFOAM_paragraph_4", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/openFOAM/#openfoam-workflow" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_4.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_4.txt new file mode 100644 index 00000000000..39e80a033cd --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_4.txt @@ -0,0 +1,35 @@ +One important aspect to keep in mind for 'offline' pre-processing is +that the domain decomposition needs to match the number of processor +cores that are used for the actual simulation, see also Domain decomposition and number of processor cores. +For post-processing you can either download the simulation results to a +local workstation, or do the post-processing (interactively) on the HPC +infrastructure, for example on the login nodes or using an interactive +session on a workernode. This may be interesting to avoid the overhead +of downloading the results locally. +Running OpenFOAM in parallel +For general information on running OpenFOAM in parallel, see +https://cfd.direct/openfoam/user-guide/running-applications-parallel/. +The -parallel option +When running OpenFOAM in parallel, do not forget to specify the -parallel option, to avoid running the same OpenFOAM +simulation $N$ times, rather than running it once using $N$ processor +cores. +You can check whether OpenFOAM was run in parallel in the output of the +main command: the OpenFOAM header text should only be included once in +the output, and it should specify a value different than '1' in the +nProcs field. Note that most pre- and post-processing utilities like +blockMesh, decomposePar and reconstructPar can not be run in +parallel. +Using mympirun +It is highly recommended to use the mympirun command when running +parallel OpenFOAM simulations rather than the standard mpirun command; +see Mympiprun for more information on mympirun. +See Basic usage for how to get started with mympirun. +To pass down the environment variables required to run OpenFOAM (which +were defined by the $FOAM_BASH script, see Preparing the environment) to each of the MPI +processes used in a parallel OpenFOAM execution, the +$MYMPIRUN_VARIABLESPREFIX environment variable must be defined as +follows, prior to running the OpenFOAM simulation with mympirun: +export MYMPIRUN_VARIABLESPREFIX=WM_PROJECT,FOAM,MPI +Whenever you are instructed to use a command like mpirun -np ..., +use mympirun ... instead; mympirun will automatically detect the +number of processor cores that are available (see also Controlling number of processes). diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_4_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_4_metadata.json new file mode 100644 index 00000000000..2d8660823ea --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_4_metadata.json @@ -0,0 +1,19 @@ +{ + "main_title": "openFOAM", + "subtitle": "Using-`mympirun`", + "source_file": "../../mkdocs/docs/HPC/openFOAM.md", + "title_depth": 3, + "directory": "openFOAM", + "links": { + "0": "https://docs.hpc.ugent.be/openFOAM/#domain-decomposition-and-number-of-processor-cores", + "1": "https://docs.hpc.ugent.be/mympirun/#mympirun", + "2": "https://docs.hpc.ugent.be/mympirun/#basic-usage", + "3": "https://docs.hpc.ugent.be/openFOAM/#preparing-the-environment", + "4": "https://docs.hpc.ugent.be/mympirun/#controlling-number-of-processes" + }, + "parent_title": "", + "previous_title": "openFOAM_paragraph_3", + "next_title": "openFOAM_paragraph_5", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/openFOAM/#using-mympirun" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_5.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_5.txt new file mode 100644 index 00000000000..f8a011e048d --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_5.txt @@ -0,0 +1,37 @@ +Domain decomposition and number of processor cores +To run OpenFOAM in parallel, you must decompose the domain into multiple +subdomains. Each subdomain will be processed by OpenFOAM on one +processor core. +Since mympirun will automatically use all available cores, you need to +make sure that the number of subdomains matches the number of processor +cores that will be used by mympirun. If not, you may run into an error +message like: +number of processor directories = 4 is not equal to the number of processors = 16 +In this case, the case was decomposed in 4 subdomains, while the +OpenFOAM simulation was started with 16 processes through mympirun. To +match the number of subdomains and the number of processor cores used by +mympirun, you should either: +- adjust the value for numberOfSubdomains in + system/decomposeParDict (and adjust the value for n accordingly + in the domain decomposition coefficients), and run decomposePar + again; or +- submit your job requesting exactly the same number of processor + cores as there are subdomains (see the number of processor* + directories that were created by decomposePar) +See Controlling number of processes to control the number of process mympirun will start. +This is interesting if you require more memory per core than is +available by default. Note that the decomposition method being used +(which is specified in system/decomposeParDict) has significant impact +on the performance of a parallel OpenFOAM simulation. Good decomposition +methods (like metis or scotch) try to limit communication overhead +by minimising the number of processor boundaries. +To visualise the processor domains, use the following command: +mympirun foamToVTK -parallel -constant -time 0 -excludePatches '(".*.")' +and then load the VTK files generated in the VTK folder into ParaView. +Running OpenFOAM on a shared filesystem +OpenFOAM is known to significantly stress shared filesystems, since a +lot of (small) files are generated during an OpenFOAM simulation. Shared +filesystems are typically optimised for dealing with (a small number of) +large files, and are usually a poor match for workloads that involve a +(very) large number of small files (see also +http://www.prace-ri.eu/IMG/pdf/IO-profiling_with_Darshan-2.pdf). diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_5_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_5_metadata.json new file mode 100644 index 00000000000..d56fe31baf2 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_5_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "openFOAM", + "subtitle": "Running-OpenFOAM-on-a-shared-filesystem", + "source_file": "../../mkdocs/docs/HPC/openFOAM.md", + "title_depth": 2, + "directory": "openFOAM", + "links": { + "0": "https://docs.hpc.ugent.be/mympirun/#controlling-number-of-processes" + }, + "parent_title": "", + "previous_title": "openFOAM_paragraph_4", + "next_title": "openFOAM_paragraph_6", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/openFOAM/#running-openfoam-on-a-shared-filesystem" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_6.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_6.txt new file mode 100644 index 00000000000..dfd61107347 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_6.txt @@ -0,0 +1,74 @@ +Take into account the following guidelines for your OpenFOAM jobs, which +all relate to input parameters for the OpenFOAM simulation that you can +specify in system/controlDict (see also +https://cfd.direct/openfoam/user-guide/controldict). +- instruct OpenFOAM to write out results at a reasonable frequency, **certainly *not*** for every single time step}; you can control this using the writeControl, writeInterval, etc. keywords; +- consider only retaining results for the last couple of time steps, + see the purgeWrite keyword; +- consider writing results for only part of the domain (e.g., a line + of plane) rather than the entire domain; +- if you do not plan to change the parameters of the OpenFOAM + simulation while it is running, set runTimeModifiable to false to avoid that OpenFOAM re-reads each + of the system/*Dict files at every time step; +- if the results per individual time step are large, consider setting + writeCompression to true; +For modest OpenFOAM simulations where a single workernode suffices, +consider using the local disk of the workernode as working directory +(accessible via $VSC_SCRATCH_NODE), rather than the shared +$VSC_SCRATCH filesystem. **Certainly do not use a subdirectory in $VSC_HOME or $VSC_DATA, since these shared filesystems are too slow +for these types of workloads. +For large parallel OpenFOAM simulations on the UGent Tier-2 clusters, consider +using the alternative shared scratch filesystem $VSC_SCRATCH_ARCANINE +(see Pre-defined user directories). +These guidelines are especially important for large-scale OpenFOAM +simulations that involve more than a couple of dozen of processor cores. +Using own solvers with OpenFOAM +See https://cfd.direct/openfoam/user-guide/compiling-applications/. +Example OpenFOAM job script +Example job script for damBreak OpenFOAM tutorial (see also +https://cfd.direct/openfoam/user-guide/dambreak): +#!/bin/bash +#PBS -l walltime=1:0:0 +#PBS -l nodes=1:ppn=4 +# check for more recent OpenFOAM modules with 'module avail OpenFOAM' +module load OpenFOAM/6-intel-2018a +source $FOAM_BASH +# purposely not specifying a particular version to use most recent mympirun +module load vsc-mympirun +# let mympirun pass down relevant environment variables to MPI processes +export MYMPIRUN_VARIABLESPREFIX=WM_PROJECT,FOAM,MPI +# set up working directory +# (uncomment one line defining $WORKDIR below) +#export WORKDIR=$VSC_SCRATCH/$PBS_JOBID # for small multi-node jobs +#export WORKDIR=$VSC_SCRATCH_ARCANINE/$PBS_JOBID # for large multi-node jobs (not on available victini) +export WORKDIR=$VSC_SCRATCH_NODE/$PBS_JOBID # for single-node jobs +mkdir -p $WORKDIR +# damBreak tutorial, see also https://cfd.direct/openfoam/user-guide/dambreak +cp -r $FOAM_TUTORIALS/multiphase/interFoam/laminar/damBreak/damBreak $WORKDIR +cd $WORKDIR/damBreak +echo "working directory: $PWD" +# pre-processing: generate mesh +echo "start blockMesh: $(date)" +blockMesh &> blockMesh.out +# pre-processing: decompose domain for parallel processing +echo "start decomposePar: $(date)" +decomposePar &> decomposePar.out +# run OpenFOAM simulation in parallel +# note: +# * the -parallel option is strictly required to actually run in parallel! +# without it, the simulation is run N times on a single core... +# * mympirun will use all available cores in the job by default, +# you need to make sure this matches the number of subdomains! +echo "start interFoam: $(date)" +mympirun --output=interFoam.out interFoam -parallel +# post-processing: reassemble decomposed domain +echo "start reconstructPar: $(date)" +reconstructPar &> reconstructPar.out +# copy back results, i.e. all time step directories: 0, 0.05, ..., 1.0 and inputs +export RESULTS_DIR=$VSC_DATA/results/$PBS_JOBID +mkdir -p $RESULTS_DIR +cp -a *.out [0-9.]* constant system $RESULTS_DIR +echo "results copied to $RESULTS_DIR at $(date)" +# clean up working directory +cd $HOME +rm -rf $WORKDIR diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_6_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_6_metadata.json new file mode 100644 index 00000000000..03f2731dc77 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/openFOAM/openFOAM_paragraph_6_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "openFOAM", + "subtitle": "Example-OpenFOAM-job-script", + "source_file": "../../mkdocs/docs/HPC/openFOAM.md", + "title_depth": 2, + "directory": "openFOAM", + "links": { + "0": "https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#pre-defined-user-directories" + }, + "parent_title": "", + "previous_title": "openFOAM_paragraph_5", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/openFOAM/#example-openfoam-job-script" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/program_examples/program_examples_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/program_examples/program_examples_paragraph_1.txt new file mode 100644 index 00000000000..3599383c404 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/program_examples/program_examples_paragraph_1.txt @@ -0,0 +1,40 @@ +Program examples { #ch:program-examples} +If you have not done so already copy our examples to your home directory by running the following command: + cp -r /apps/gent/tutorials/Intro-HPC/examples ~/ +~(tilde) refers to your home directory, the directory you arrive by default when you login. +Go to our examples: +cd ~/examples/Program-examples +Here, we just have put together a number of examples for your +convenience. We did an effort to put comments inside the source files, +so the source code files are (should be) self-explanatory. +1. 01_Python +2. 02_C_C++ +3. 03_Matlab +4. 04_MPI_C +5. 05a_OMP_C +6. 05b_OMP_FORTRAN +7. 06_NWChem +8. 07_Wien2k +9. 08_Gaussian +10. 09_Fortran +11. 10_PQS +The above 2 OMP directories contain the following examples: +| C Files | Fortran Files | Description | +|------------------|------------------|---------------------------------------| +| omp_hello.c | omp_hello.f | Hello world | +| omp_workshare1.c | omp_workshare1.f | Loop work-sharing | +| omp_workshare2.c | omp_workshare2.f | Sections work-sharing | +| omp_reduction.c | omp_reduction.f | Combined parallel loop reduction | +| omp_orphan.c | omp_orphan.f | Orphaned parallel loop reduction | +| omp_mm.c | omp_mm.f | Matrix multiply | +| omp_getEnvInfo.c | omp_getEnvInfo.f | Get and print environment information | +| omp_bug | omp_bug| omp_bug | omp_bug| Programs with bugs and their solution | +Compile by any of the following commands: +| Language | Commands | +|--------------|----------------------------------------| +| C: | icc -openmp omp_hello.c -o hello | +| | pgcc -mp omp_hello.c -o hello | +| | gcc -fopenmp omp_hello.c -o hello | +| Fortran: | ifort -openmp omp_hello.f -o hello | +| | pgf90 -mp omp_hello.f -o hello | +| | gfortran -fopenmp omp_hello.f -o hello | diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/program_examples/program_examples_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/program_examples/program_examples_paragraph_1_metadata.json new file mode 100644 index 00000000000..02b1523bdc4 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/program_examples/program_examples_paragraph_1_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "program_examples", + "subtitle": "Program-examples", + "source_file": "../../mkdocs/docs/HPC/program_examples.md", + "title_depth": 1, + "directory": "program_examples", + "parent_title": "", + "previous_title": null, + "next_title": "program_examples_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/program_examples/#program-examples" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/program_examples/program_examples_paragraph_2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/program_examples/program_examples_paragraph_2.txt new file mode 100644 index 00000000000..a43f15893e7 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/program_examples/program_examples_paragraph_2.txt @@ -0,0 +1 @@ +Be invited to explore the examples. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/program_examples/program_examples_paragraph_2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/program_examples/program_examples_paragraph_2_metadata.json new file mode 100644 index 00000000000..baf30f23ad3 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/program_examples/program_examples_paragraph_2_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "program_examples", + "subtitle": "Program-examples", + "source_file": "../../mkdocs/docs/HPC/program_examples.md", + "title_depth": 1, + "directory": "program_examples", + "parent_title": "", + "previous_title": "program_examples_paragraph_1", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/program_examples/#program-examples" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/quick_reference_guide/quick_reference_guide_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/quick_reference_guide/quick_reference_guide_paragraph_1.txt new file mode 100644 index 00000000000..4ed8b90156e --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/quick_reference_guide/quick_reference_guide_paragraph_1.txt @@ -0,0 +1,37 @@ +HPC Quick Reference Guide +Remember to substitute the usernames, login nodes, file names, ...for +your own. +| Login | | +|-------------------|-----------------------------------------| +| Login | ssh vsc40000@login.hpc.ugent.be | +| Where am I? | hostname | +| Copy to HPC | scp foo.txt vsc40000@login.hpc.ugent.be: | +| Copy from HPC | scp vsc40000@login.hpc.ugent.be:foo.txt | +| Setup ftp session | sftp vsc40000@login.hpc.ugent.be | +| Modules | | +|----------------------------|-----------------------| +| List all available modules | Module avail | +| List loaded modules | module list | +| Load module | module load example | +| Unload module | module unload example | +| Unload all modules | module purge | +| Help on use of module | module help | +| Command | Description | +|-----------------------------------------------|---------------------------------------------------------| +| qsub script.pbs | Submit job with job script script.pbs | +| qstat 12345 | Status of job with ID 12345 | + | qstat -n 12345 | Show compute node of job with ID 12345 | +| qdel 12345 | Delete job with ID 12345 | +| qstat | Status of all your jobs | +| qstat -na | Detailed status of your jobs + a list of nodes they are running on | + | qsub -I | Submit Interactive job | +| Disk quota | | +|-----------------------------------------------|-------------------------------------------------| + | Check your disk quota | see https://account.vscentrum.be | + | Disk usage in current directory (.) | du -h | +| Worker Framework | | +|-----------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------| +| Load worker module | module load worker/1.6.12-foss-2021b Don't forget to specify a version. To list available versions, use module avail worker/ | +| Submit parameter sweep | wsub -batch weather.pbs -data data.csv | +| Submit job array | wsub -t 1-100 -batch test_set.pbs | +| Submit job array with prolog and epilog | wsub -prolog pre.sh -batch test_set.pbs -epilog post.sh -t 1-100 | \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/quick_reference_guide/quick_reference_guide_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/quick_reference_guide/quick_reference_guide_paragraph_1_metadata.json new file mode 100644 index 00000000000..577fa61744b --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/quick_reference_guide/quick_reference_guide_paragraph_1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "quick_reference_guide", + "subtitle": "HPC-Quick-Reference-Guide", + "source_file": "../../mkdocs/docs/HPC/quick_reference_guide.md", + "title_depth": 1, + "directory": "quick_reference_guide", + "links": { + "0": "https://account.vscentrum.be" + }, + "parent_title": "", + "previous_title": null, + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/quick_reference_guide/#hpc-quick-reference-guide" +} \ No newline at end of file diff --git a/scripts/README.md b/scripts/README.md index eed5a73e4d5..a88bd42cc46 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,3 +1,4 @@ Scripts that can be used to automatically generate markdown files, can be found here. -* [`available_software`](available_software): script to generate overview of available environment modules; \ No newline at end of file +* [`available_software`](available_software): script to generate overview of available environment modules; +* [`chatbot_preprocessor`](HPC_chatbot_preprocessor): script to generate input files for the chatbot; \ No newline at end of file