====== SLURM usage guide ======
----
**If you are completely new to scientific computing, read this:**
Please keep in mind that simply throwing a serial program without provisions for parallelization at the cluster will not make it run faster. In fact, it will often run SLOWER, since cpu cores on machines specialized in scientific computing usually run at lower clock frequencies than workstation cpus.
There are many commercial and open source software packages that are already very well parallelized, **but you'll definitely need to know how to use the parallel capabilities of your software or programming language**. Requesting lots of nodes, cpu cores, memory and time for a program that will use only one single cpu will only keep you and all other users waiting.
So **start small and simple**. There's no use waiting for a 10-node-job just to find out it immeadiately crashes, so test with a 1-node-1-task-job that requests only 10 minutes first, then moderately make it bigger and check that e.g. your parallel program delivers reasonable results. When you are sure you understand what you do and when your test case works reliably, when you see a decrease in run time when you add ressources --- then go for it.
===== Why use a cluster at all? =====
The reason you want to use the cluster is probably the computing resources it provides. With about several hundred people using the compute cluster for their research every year, there has to be an instance organizing and allocating these resources. This instance is called the batch system ("scheduler", "resource manager", in the LUH-cluster: "SLURM"). The batch system sorts the ressource requests ("batch jobs") it gets from the users according to ressource availability and priority rules. When the priority of a job is sufficiently high to start, it gets scheduled on the requested ressources (usually some compute node(s)) and starts. Requests to the batch system are usually made by submitting a text file containing (bash-) instructions about what to do (the "batch job"), or by requesting an "interactive" batch job from the command line that puts you directly into a command shell on a compute node (or a set of compute nodes) to work with, or, as a third option, by the Open OnDemand portal ("OOD") that is available on https://login.cluster.uni-hannover.de which translates the settings you enter via the web browser gui into a text file to again submit as a batch job to the batch system.
===== Parallelization Basics =====
Batch jobs can roughly be divided into several categories:
* serial (also: single-threaded, single-task) batch jobs; they run just like normal programs/scripts. Usually, no changes/adaptions are needed. The time needed to complete such a job almost entirely depends on the speed of a single cpu core, and no scaling is achieved. As mentioned above, starting a serial program on a larger machine will NOT make it run faster, and since the compute servers in the cluster usually combine lots of cpu cores on each cpu socket, they actually may run even slower than cores on a smaller workstation that has fewer cores, due to the total heat generated by each chip that needs to be dissipated. So think about the size and characteristics of your workload. In the "real world", smaller loads over short distances are better transported using a fast car, while large ones that travel around the world need a big container ship. Similar considerations are valid for workloads on computers, in particular, if the software is not parallelized.
* parallel jobs; these in turn can be distinguished by the kind of problems they treat and how they attempt to achieve scaling ("reaching results quicker"). Problem are either "trivially parallel computations" or not:
* trivially parallel problems can be solved by simply starting as many tasks as needed / as possible, and each task will run happily minding its own business, until it achieves a partial result in the end, which is then combined with all the other results of the other tasks to get the result of the whole job. Luckily, many problems can be computed in this way.
* non-trivial problems are those that usually need to exchange data //during// computation, for example when finishing a simulation time step to update the (so-called) "ghost"-borders of all the simulation subdomains the complete simulation region has been decomposed into.
* The other main distinction of parallel jobs is //how// they do it. The two main variants here are:
* OpenMP jobs (shared-memory-processing, SMP, single-node, multi-threaded, multi-processing, can typically scale to the larges compute nodes available) usually run on ONE node only, using multiple threads or processes, but sharing memory. So the software needs some logic that specifies which parts of the program (e.g. loops) should run in parallel, but synchronization between threads is automatic. These programs typically are linked to an OpenMP library during compilation, so the node itself usually needs to have some libraries installed, but parallelization is relatively easy and low-effort. The software must ensure that only one process updates a specific memory location at the same time, but it can rely on having the same memory contents. Beware, though, that while many application programs are parallelized this way, and nowadays compute nodes may contain many cpu cores, scaling still may be quite limited depending on the specific kind of problem and how much effort has been spent to parallelize the regions of the software that should do parallel computations. So, while some software packages achieve very good scaling up to over 100 cpu cores on big servers, others already hit their limits when using more than 4 or 8 cpus. There's many hardware-factors limiting performance, too, like the number of cache levels, CPU cache sizes, memory bandwidth, NUMA architecture etc, and of course I/O.
* MPI (message-passing-interface, possible multi-node, multi-processing, can scale up to millions of cpus); software parallelized using MPI is usually able to achieve the highest scaling, but the cost is that the software must explicitly specify which parts of the program run in parallel and how data / results are updated, which task does what, and how the simulation is kept in sync. So the scaling here also depends on the genius of the person writing the software and their knowledge about specific hardware features. Each MPI-task (called a "rank") is highly independent of the others, and it needs to explicitly communicate its results to the other tasks whenever there's a need for that, since each task uses their own memory that only they can access.
Hint: to avoid an easy understanding of complicated things, someone thought it would be a good idea to name one of the several MPI libraries available "OpenMPI". Do not fall for this trap. OpenMPI is one specific implementation of the MPI programming interface (there are many others called IntelMPI, MVAPICH, IBMMPI, ...) that uses message-passing between independent tasks to achieve a high degree of parallelization. OpenMP, on the other hand, is a general term for a completely different programming interface that is using compiler-directives and which has NOT much to do with MPI. So OpenMP is NOT MPI, while OpenMPI is MPI, but NOT OpenMP.
===== The SLURM Workload Manager =====
The software that decides which job to run when and where in the cluster is called SLURM. SLURM (**S**imple **L**inux **U**tility for **R**esource **M**anagement) is a free open-source batch scheduler and resource manager that allows users to run their jobs on the LUIS compute cluster. It is a modern, extensible batch system that is installed on many clusters of various sizes around the world. This chapter describes the basic tasks necessary for submitting, running and monitoring jobs under the SLURM Workload Manager on the LUIS cluster. Detailed information about SLURM can be found on the official SLURM [[https://slurm.schedmd.com/documentation.html|website]].
Here are some of the most important commands to interact with SLURM:
* **sbatch** - submit a batch script
* **salloc** - allocate compute resources
* **srun** - allocate compute resources and launch job-steps
* **squeue** - check the status of running and/or pending jobs
* **scancel** - delete jobs from the queue
* **sinfo** - view intormation abount cluster nodes and partitions
* **scontrol** - show detailed information on active and/or recently completed jobs, nodes and partitions
* **sacct** - provide the accounting information on running and completed jobs
* **slurmtop** - text-based view of cluster nodes' free and in-use resources and status of jobs
Some usage examples for these commands are provided below. As always, you can find out more using the manual pages on a terminal/console on the system (like ''man squeue'') or on the SLURM manuals' [[https://slurm.schedmd.com/man_index.html|website]].
===== Partitions =====
Compute nodes with similar hardware attributes (like e.g. the same cpu) in the cluster are usually grouped in partitions. Each partition can be regarded as somewhat independent from others. A batch job can be submitted in such a way that it can run on one of several partitions, and a compute node may also belong to several partitions simultaneously to facilitate selection. Jobs are allocated resources like cpu cores, memory and time within a single partition for executing tasks on the cluster. A concept called “job steps” is used to execute several tasks simultaneously or sequentially within a job using the ''srun'' command.
The table below lists the currently defined partitions and their parameters/constraints:
^ Part of cluster ^ Max Job Runtime ^ Max Nodes Per Job ^ Max CPUs per User ^ Default Runtime ^ Default Memory per CPU ^ Shared Node Usage ^
| amo, dumbo, haku, lena, taurus, ... (generic) | 200 hours | | 800 | 24 hours | 4000 MB | yes |
| gpu nodes | 48 hours | 1 | | 1 hour | 1600 MB | yes |
To keep things fair, control job workload and keep SLURM responsive, we enforce some additional restrictions:
^ SLURM limits ^ Max number of jobs running ^ Max number of jobs submitted ^
| per user | 64 | 500 |
| cluster-wide | 10000 | 20000 |
Based on available resources and when still able to maintain a fair balance between all users' needs, we may sometimes also consider requests for a higher priority for a short time, which may be submitted to [[cluster-help@luis.uni-hannover.de|]]. You should include an explanation for what period of time you need which kind of priority, and of course why we should consider your request regarding the fact that usually all other users want priority, too.
To list job limits relevant for you, use the ''sacctmgr'' command:
sacctmgr -s show user
sacctmgr -s show user format=user,account,maxjobs,maxsubmit,maxwall,qos
Up-to-date information on ALL available nodes:
sinfo -Nl
scontrol show nodes
Information on partitons and their configuration:
sinfo -s
scontrol show partitions
The ''clusterinfo'' command (Python script) retrieves real-time information about node and partition configurations, resource (CPU/GPU) usage, and user access rights to resources through native SLURM commands and displays the data in a structured format for easier interpretation.
It shows which nodes are accessible by all users and which are reserved for specific research groups with exclusive access during configured times (see [[https://www.luis.uni-hannover.de/de/services/computing/forschungscluster-housing/|Forschungscluster-Housing]]). By executing ''clusterinfo -l'', your configured SLURM limits (such as the maximum number of running and pending jobs, maximum wall clock time, etc.) will also be displayed. For a list of available options and their descriptions, run ''clusterinfo -h''.
===== Interactive jobs =====
Please note: when you have a //non-interactive// (standard) reservation/running job on a node or a set of nodes, you may //also// directly open additional shell(s) to that node(s) coming from a login node, e.g. for watching/debugging/changing what happens. But beware: you will get kicked out as soon as your job finishes.
Batch submission is the most common and most efficient way to use the computing cluster. Interactive jobs are also possible; they may be useful for things like:
* working with an interactive terminal or GUI applications like R, iPython, ANSYS, MATLAB, etc.
* software development, debugging, or compiling
You can start an interactive session on a compute node using the SLURM ''salloc'' command. The following example submits an interactive job that requests 12 tasks (this corresponds to 12 MPI ranks) on two compute nodes and 4 GB memory per CPU core for an hour:
[user@login02 ~]$ salloc --time=1:00:00 --nodes=2 --ntasks=12 --mem-per-cpu=4G --x11
salloc: slurm_job_submit: set partition of submitted job to amo,tnt,gih
salloc: Pending job allocation 27477
salloc: job 27477 queued and waiting for resources
salloc: job 27477 has been allocated resources
salloc: Granted job allocation 27477
salloc: Waiting for resource configuration
salloc: Nodes amo-n[001-002] are ready for job
[user@amo-n001 ~]$
The option ''%%--%%x11'' sets up X11 forwarding on the first(master) compute node enabling the use of graphical applications.
**Note:** Unless you specify a cluster partition explicitly, all partitions that you have access to will be available for your job.
**Note:** If you do not explicitly specify memory and time parameters for your job, the corresponding default values for
the cluster partition to which the job will be assigned will be used. To find out the default time and memory settings for a partition, e.g. ''amo'',
look at the ''DefaultTime'' and ''DefMemPerCPU'' values in the ''scontrol show partitions amo'' command output.
**Note:** In case you get an error message like ''srun: Warning: can't honor %%--%%ntasks-per-node set to X which doesn't match the requested tasks YY with the number of requested nodes ZZ. Ignoring'', check (using ''set | grep SLURM_N'' within the job shell, for example) that your request has been honored despite the message, and then ignore the message. :-)
Once the job starts, you will get an interactive shell on the first compute node (''amo-n001'' in the example above) that has been assigned to the job, where you can interactively spawn your applications. The following example compiles and executes the MPI ''Hello World'' program (save the source code to the file ''hello_mpi.c''):
#include "mpi.h"
#include
int main (int argc, char** argv) {
int ntasks, taskid, len;
char hostname[MPI_MAX_PROCESSOR_NAME];
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD,&ntasks);
MPI_Comm_rank(MPI_COMM_WORLD,&taskid);
MPI_Get_processor_name(hostname, &len);
printf ("Hello from task %d of %d on %s\n", taskid, ntasks, hostname);
MPI_Finalize();
}
[user@amo-n001 ~]$ module load GCC/9.3.0 OpenMPI/4.0.3
[user@amo-n001 ~]$ mpicc hello_mpi.c -o hello_mpi
[user@amo-n001 ~]$ srun --ntasks=6 --distribution=block hello_mpi
Hello from task 0 of 6 on amo-n001
Hello from task 1 of 6 on amo-n001
Hello from task 2 of 6 on amo-n001
Hello from task 3 of 6 on amo-n001
Hello from task 4 of 6 on amo-n001
Hello from task 5 of 6 on amo-n002
**Note:** If you want to run a parallel application using Intel MPI Library (e.g by loading the module ''impi/2020a'') then provide the ''srun'' command with an additional option ''%%--%%mpi=pmi2''
**Note:** Environment variables set on the login node from which the job was submitted are not passed to the job.
The interactive session is terminated by typing ''exit'' on the shell:
[user@amo-n001 ~]$ exit
logout
salloc: Relinquishing job allocation 27477
Alternatively you can use the ''srun %%--%%pty $SHELL -l'' command to interactively allocate compute resources, e.g.
[user@login02 ~]$ srun --time=1:00:00 --nodes=2 --ntasks=12 --mem-per-cpu=4G --x11 --pty $SHELL -l
srun: slurm_job_submit: set partition of submitted job to amo,tnt,gih
[user@amo-n004 ~]$
At this point, we would like to note that SLURM differentiates between ''%%--%%ntasks'', which may roughly be translated into the number of (independent) MPI-ranks or instances of a job, and ''%%--%%cores-per-task'', which may translate into the number of (OpenMP) threads. MPI-jobs usually request ''%%--%%ntasks'' larger than one, while OpenMP-jobs may request ''%%--%%ntasks=1'' and ''%%--%%cores-per-task'' higher than one.
If you want to run your jobs on nodes with a specific CPU type, you can request them using the SLURM option ''%%--%%constraint=CPU_ARCH:'', where '''' can currently have the following values: ''sse'', ''avx'', ''avx2'', and ''avx512''.
To check the available CPU architectures for different SLURM partitions and nodes, you can use the command ''clusterinfo -n -i''.
If your job can run on nodes with any of multiple CPU types, you can specify them using the following syntax: ''%%--%%constraint=[CPU_ARCH:,CPU_ARCH:,...].''
===== Submitting a batch script =====
A SLURM job submission file for your job (a "batch script") is a shell script with a set of additional directives that are only interpreted by the batch system (Slurm) at the beginning of the file. These directives are marked by starting the line with the string ''#SBATCH'', so the batch system knows that the following parameters and commands are not just a comment (which the ''#'' character otherwise would imply). The shell (the command line interpreter of Unix) usually ignores everything that follows a ''#'' character. But at the beginning of your file, the Slurm commands used to submit a batch script will also check whether the ''#'' character is immediately followed by ''SBATCH''. If that is the case, the batch system will interpret the following characters as directives. Processing of these directives stops once the first non-comment non-whitespace line has been reached in the script. The very first line of your script usually should read ''#!/bin/bash'' - ask Wikipedia for the meaning of "Shebang (Unix)" in case you want to understand what this is for.
Valid directives can be found using the command ''man sbatch''. In principle, you may write almost any option that you could feed to sbatch at the command line as a #SBATCH-line in your script.
A suitable batch script is usually submitted to the batch system using the ''sbatch'' command.
==== An example of a serial job ====
The following is an example of a simple serial job script (save the lines to the file ''test_serial.sh'').
**Note:** change the ''#SBATCH'' directives to your use case where applicable.
#!/bin/bash -l
#SBATCH --job-name=test_serial
#SBATCH --ntasks=1
#SBATCH --mem-per-cpu=2G
#SBATCH --time=00:20:00
#SBATCH --constraint=[CPU_ARCH:avx512|CPU_ARCH:avx2]
#SBATCH --mail-user=user@uni-hannover.de
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --output test_serial-job_%j.out
#SBATCH --error test_serial-job_%j.err
# Change to my work dir
# SLURM_SUBMIT_DIR is an environment variable that automatically gets
# assigned the directory from which you did submit the job. A batch job
# is like a new login, so you'll initially be in your HOME directory.
# So it's usually a good idea to first change into the directory you did
# submit your job from.
cd $SLURM_SUBMIT_DIR
# Load the modules you need, see corresponding page in the cluster documentation
module load my_modules
# Start my serial app
# srun is needed here only to create an entry in the accounting system,
# but you could also start your app without it here, since it's only serial.
srun ./my_serial_app
To submit the batch job, use
sbatch example_serial_slurm.sh
**Note:** as soon as compute nodes are allocated to your job, you can establish an ''ssh'' connection from the login machines to these nodes.
**Note:** if your job oversteps the resource limits that you have defined in your ''#SBATCH'' directives, the job will automatically be killed by the SLURM server. This is particularly the case when you try to use more memory than you allocated, which results in an OOM (out-of-memory) -event.
The table below shows frequently used sbatch options that can either be specified in your job script with the ''#SBATCH'' directive or on the command line. Command line options override options in the script. The commands ''srun'' and ''salloc'' accept the same set of options. Both long and short options are listed.
^Options ^ Default Value ^Description |
|''%%--%%nodes='' or ''-N '' | 1 |Number of compute nodes |
|''%%--%%tasks='' or ''-n '' | 1 |Number of tasks to run |
|''%%--%%cpus-per-task='' or ''-c '' | 1 |Number of CPU cores per task |
|''%%--%%ntasks-per-node='' | 1 |Number of tasks per node |
|''%%--%%ntasks-per-core='' | 1 |Number of tasks per CPU core |
|''%%--%%mem-per-cpu='' | partition dependent |memory per CPU core in MB |
|''%%--%%mem='' | partition dependent |memory per node in MB |
|''%%--%%gres=gpu::'' | - |Request nodes with GPUs; may be omitted (thus: ''%%--%%gres=gpu:'') |
|''%%--%%time=