The queue system used for the public and open-project "burros" is Slurm, widely used in many research institutes and supercomputing centres. You can find a general guide on how to use Slurm at IAC machines here.
Below we describe the default configuration and some commands specific to the Slurm installation in the IAC public "burros". Open-project "burros" share most of the configuration of a public "burro", but with some usage conditions, which are described in the section on open-project burros below.
Default Slurm configuration in a public "burro"
Note
This configuration can be modified in project "burros" to take into account the needs of each research group.
When a workstation is shared amongst several users it is easy to oversubscribe it (e.g. to run more processes than available cores), which makes all applications to perform worse and it is usually detrimental to all users. To avoid this problem, the default Slurm configuration in a "burro" is as follows:
Limited CPU power
A default ssh or graphical session is limited to the CPU power equivalent to 3/4 of a core. This should be enough for editing files, checking on the state of the machine, etc. but certainly not enough for running any intensive computer program. For this, you will need to use the Slurm queueing system, either using an interactive session or submitting a batch job.
Important
Slurm accounts are independent of the Linux system accounts, so before you use Slurm you will need to get a Slurm account:
if using a project burro, contact the PI of the project.
if using a public or an open-project burro, you can activate an account by yourself with the
slurm_addmecommand. The following example shows how a first call to thesintercommand (see section Getting and interactive session below) is not successful because the account foruserwas not yet created.slurm_addmecreates the account and the followingsintercommand is successful.
[user@burro ~]$ sinter -n 2
salloc: error: Job submit/allocate failed: Invalid account or account/partition combination specified
[user@burro ~]$ slurm_addme
Associations =
U = user A = ddgroup C = burro
Non Default Settings
[user@burro ~]$ sinter -n 2
salloc: Granted job allocation 127003
salloc: Nodes burro are ready for job
(sinter) [user@burro ~]$
Partitions configuration
In order to have a reasonable balance of batch/interactive jobs, we configure two "partitions" (or "queues").
batch, meant for long heavy-computation jobs, with the following limits:Max CPUs: number of hardware threads of node (if hardware threads are disabled in the node, this is the same as the number of cores)
Max time: 2 days (QOS:
normal) or 7 days (QOS:long). With Slurm you can specify a Quality of Service (QOS) for each job. We have configured thebatchpartition with two QOSs:normal(default QOS), with max time of jobs of 2 days;long, with max time of jobs of 7 days, but with a greatly reduced priority when compared to thenormalQOS. If possible, try to avoid using thelongQOS (it is much better if you can cut your job in smaller parts or if you can prepare your application to frequently create checkpoints that can be used to restart your application with the state saved by a previous run). If really necessary, request this QOS with--qos=long.
interactive, meant for lighter interactive jobs, with the following limits:Max CPUs per job: number of hardware threads of node
Max CPUs per partition: 2 * (number of hardware threads of node) [via oversubscription]
Max time: 8 hours
At any time you can check the actual configuration of the partitions
with the sinfo command. For example:
$ sinfo -O SocketCoreThread,Partition,Maxcpuspernode,time
S:C:T PARTITION MAX_CPUS_PER_NODE TIMELIMIT
2:28:2 batch* 112 7-00:00:00
2:28:2 interactive 112 8:00:00
Note
While having fine-grained control of where each process runs is not necessary for many jobs, those running parallel applications and advanced users might want to read our notes in section CPU Management).
Important
It might seem obvious, but remember that the "interactive" queue is for "interactive" jobs only!
Acceptable use cases for the interactive queue are, for example: if you need to test whether your code compiles OK with different compilation options; if you are writing code and need to test if it works as you develop it; if you are debugging code; if you need to run a GUI application, etc.
Non-acceptable use cases are running long CPU-intensive jobs that do not require your interaction in a long time (10 minutes or more).
This is so because jobs in these two partitions can overlap, so batch
jobs can run alongside interactive ones (and this assumes that
interactive jobs are not as computation intensive as batch ones, and
allows a controlled oversubscription of the workstation).
In total, up to three times the number of hardware threads could be
allocated at the same time (one in the batch partition, and two in the
interactive partition). This works OK only if the jobs in the
interactive partition are light jobs, GUI apps interactions, Python
interactive sessions, etc.
Don't abuse the system by submitting long CPU-intensive jobs in the "interactive" queue or the system administrators will show you a yellow (or even a red) card!
Tracking Memory usage
In a shared "burro", memory usage also needs to be tracked, so that we ensure that a job cannot saturate the available RAM (which can negatively impact on other jobs' performance and even bring the whole system down). The way we track memory usage is as follows:
Note
Memory limitation is activated by default in all public and open-project "burro"s, but not in project ones. If you want your "burro" to have this limitation, please let us know.
by default, when submitting a job to Slurm, each allocated CPU will also be allocated a default RAM amount. Since we have to guarantee that a machine at full capacity does not consume more memory than available, and given that up to three times the number of hardware threads could be allocated at any given time (see above), the default RAM allocated per CPU is:
\[(1/3) Total\_memory / N_\mathrm{threads}\]As an example, one of our "burro"s has 32 hardware threads, and a total of 256 GB of RAM, so the default RAM allocated per CPU is 2.67 GB, which can be checked with the following command (output is in MB):
$ scontrol show config | grep DefMemPerCPU DefMemPerCPU = 2667
if more/less memory is required, you can use the
--mem-per-cpuor the--memoptions to specify the required amount.For example, if we only need one CPU, but we require 60 GB of memory, the default value of 2.67 GB will not be enough, and the job will be killed when this value is exceeded. To avoid that, we can simply submit a job with a batch script similar to the one below (see section Batch jobs below for more information on batch scripts).
#!/bin/bash ############################# #SBATCH -J test #SBATCH -n 1 #SBATCH -t 00:15:00 #SBATCH --mem=60G ############################# ./your_job
Sometimes, it is perhaps easier to specify the amount of memory requested per process (for example, in MPI parallel applications), so in that case we could use a batch script similar to:
#!/bin/bash ############################# #SBATCH -J test #SBATCH -n 10 #SBATCH -t 00:15:00 #SBATCH --mem-per-cpu=6G ############################# srun ./your_mpi_job
Tracking GPU usage
In those "burro"s that have GPUs, their usage is also tracked with Slurm, so that a user can have exclusive access to some or all the GPUs. Slurm is configured in a way that jobs requesting a GPU have a much larger priority than other jobs, but non-GPU jobs can also use the "burro". In the Introduction section you can see the table of currently available "burro"s, and whether they have GPUs.
The way that GPU usage and tracking is integrated with Slurm in a "burro" is as follows:
You will be able to use the GPU card only from inside a Slurm job, either interactive or batch. As a matter of fact, outside of Slurm it will look like the "burro" has no GPUs (e.g. the
nvidia-smicommand will report aFailed to initialize NVML: Unknown Error).You request a GPU by adding the Slurm option
--gres=gpu. Once allocated, you will have exclusive access to the GPU for the duration of the job.
You can get more information and examples of use of the GPUs at the burros here.
Integration of Slurm with periodic reboots
Due to security considerations the IAC "burros" need to be rebooted regularly whenever security patches for any of the Ubuntu packages are published. The exact date of these reboots is not fixed, but we can expect in the order of one or two reboots per month.
For machines without Slurm these reboots can be quite disruptive since the reboots will kill all running processes (messages are broadcasted to all open terminals in each "burro" to be rebooted, but it is quite easy to miss them if you are executing long-running processes).
"Burros" using Slurm nicely integrate the information about periodic reboots so as to make sure that no jobs will be running when the workstation is rebooted. See the "note" below to understand how this feature works.
Note
Whenever a reboot is scheduled, we automatically create a Slurm reservation of 30 minutes around the exact time when the reboot will take place. For example, if a reboot is scheduled for tomorrow at 07:00, a Slurm reservation of 30 minutes will be placed around that time and Slurm will make sure that no jobs can be running during that reservation.
So, if you submit now a 48 hours job, it will not start running before the reboot because if it were to use the full allocated 48 hours it would clash with the reservation of tomorrow. In this case, you can either reduce the allocation of the job so that it will end before the reservation begins, or you can leave the job in the queue and it will start running after the reboot whenever Slurm can find a suitable allocation slot for it.
You can see if there are planned reservations by using the command:
scontrol show res
Also, if you submit a job that clashes with a planned reservation,
you will see when using the squeue command that the job stays in the queue
in the PENDING (PD) state, and that the NODELIST (REASON) column reads
something like (ReqNodeNotAvail, Reserved for maintenance).
Important
As mentioned below, access to network-shared resources can give troubles in certain cases due to Kerberos tickets expiring. Integrated with Slurm we have a plugin called AUKS which takes care of renewing your Kerberos tickets for a certain time period, which can alleviate most of the Kerberos+Slurm problems.
But the AUKS plugin "magic" does not survive a reboot, so if your jobs
make use of any network-shared resources (like /home or /net) and
they are scheduled to run after a reboot, they will fail because they will
not have access to those resources. To avoid these problems, try to move your
data to the local disks of the burros.
Integration of Slurm (and HTCondor) with the CPD temperature sensor
To protect the compute nodes (the burros) from overheating in case of an air conditioning failure or any other incident in the Data Centre (CPD, Centro de Procesamiento de Datos), the temperature in the CPD is monitored continuously and fed into both of our batch systems.
A simple three-state model is used, based on two temperature thresholds (currently 28°C and 30°C):
OK — temperature at or below 28°C: everything operates normally.
SOFT — temperature above 28°C: no new jobs are accepted. In Slurm, the nodes are set to
DRAINwith reasontemperature_soft, so any currently running jobs are allowed to finish. In HTCondor, thestartddaemons are stopped, which terminates running Condor jobs quickly.HARD — temperature above 30°C: in addition to the SOFT actions, any Slurm jobs still running are also killed by setting the nodes
DOWNwith reasontemperature_hard.By default these jobs are requeued and will start again on their own once the temperature drops back to OK and the nodes are resumed. Up to 5 automatic requeues per job are allowed; beyond that, the job is held with reason
JobHoldMaxRequeue.If for any reason you would prefer that your job is not requeued in this case, you can add
#SBATCH --no-requeueto you submission script, and then it will end up asNODE_FAILand it should be resubmitted manually.Interactive sessions (
sinter) cannot be requeued by their nature — the session is tied to your terminal, and you will see it terminate.
Once the temperature situation is over and the temperature is again ≤28ºC, then both systems go back to normal operation and jobs will be accepted again.
State transitions due to temperature changes in the CPD trigger automatic notifications in the #computing/burros Zulip channel.
Getting an interactive session
Before the installation of
Slurm, users
ssh-ed to the "burros" and executed jobs interactively in that
session. As stated above, this will now (by default) give you only the
CPU power equivalent to 3/4 of a core. In order to obtain a more
powerful interactive session you need to make use of the sinter
command, specifying the number of cores requested and the time limit for
the session. The syntax for sinter is:
sinter -n <n_cores> -t <hh:mm:ss>
For example, if you want to request a 10 core interactive session for
two and a half hours, you could do (note that (sinter) is added to
your command line to remind you that you are now in a Slurm interactive
session):
[user@burro ~]$ sinter -n 10 -t 02:30:00
salloc: Granted job allocation 127221
salloc: Nodes burro are ready for job
(sinter) [user@burro ~]$
Note
Do not ask for more cores than you need, as other people may be waiting to use the "burro".
After the requested time has passed, the session will expire and
sintermust be called again.The shorter the allocation time, the more likely that your job can start right away.
If you finish working, type
exitto quit the interactive session, so you can make the cores available for other users.For advanced users, note that
sinteris just a wrapper around the standard salloc Slurm command.
MPI Jobs
If you want to test MPI jobs while in the interactive session, you just
need to use the srun command, while specifying the number of tasks
you want to use (less or equal to the number of cores you requested for
the interactive session). So, for example, after requesting the 10 core
interactive session as per the example above, you could simply run a 4
core MPI job by simply running:
(sinter) [user@burro ~]$ srun -n 4 ./your_mpi_application
Note
Not specifying the number of tasks to use and just executing srun
./your_mpi_application is equivalent to requesting all the cores requested
for the interactive session.
Warning
If your application has been compiled with Intel MPI, you should add
--mpi=pmi2 to your srun command (for example, srun -n 4 --mpi=pmi2
./your_mpi_application).
Tip
Running a Python script using MPI would simple require srun python
script.py.
Running IPython in parallel (using MPI) in an
interactive session is doable (see IPython Parallel), but certainly
not straightforward. If you just execute ipython in an interactive
session and then try to load MPI with mpi4py (either directly or via
another package that you use) you will receive an error message that says
The application appears to have been direct launched using "srun", but OMPI
was not built with SLURM's PMI support and therefore cannot execute. If you
really need this, please do get in touch with us and we will try to get it
working.
For debugging purposes, you can launch IPython in an interactive session with
srun -n 1 --pty ipython. This will give you only one MPI rank (the same
as if you were to launch IPython without Slurm), but your code will be able
to load the MPI libraries without errors and if it uses other
multi-processing libraries it will be able to use the number of cores
requested for the interactive session.
Batch jobs
For general information about Slurm and how to submit batch jobs, please refer to the IAC-wide guide to Slurm.
Below we provide some information applicable only to the "burros".
Getting job notifications by e-mail
If you want to receive notification by e-mail of when your job begins and ends, add the following to the "batch script" (see Batch jobs).
#SBATCH --mail-type=BEGIN,END
#SBATCH --mail-user=<e-mail>
Warning
Please note that mails will only be sent to IAC mail addresses.
As an example, when your job ends, you will receive an e-mail containing information like the following:
######################## JOB EFFICIENCY REPORT ########################
# Job ID: 127267
# Cluster: <clustername>
# User/Group: <username>/<groupname>
# State: COMPLETED (exit code 0)
# Cores: 1
# CPU Utilized: 00:00:11
# CPU Efficiency: 84.62% of 00:00:13 core-walltime
# Wall-clock time: 00:00:13
# Memory Utilized: 1.30 MB
#######################################################################
MPI Jobs
Slurm and OpenMPI in the "burros" are nicely integrated, so when
submitting a batch MPI job you simply have to use the command srun,
without worrying about the number of cores needed, which is taken
directly from the Slurm allocation, or the Process Management Interface
(PMI). As a basic example, your submission script could be simply as
follows:
#!/bin/bash
#############################
#SBATCH -J your_job
#SBATCH -n 4
#SBATCH -t 00:01:00
#SBATCH -o %j.out
#SBATCH -e %j.err
#SBATCH -D .
#############################
srun ./your_mpi_application
Warning
If your application has been compiled with Intel MPI, you should add
--mpi=pmi2 to your srun command (for example, srun -n 4 --mpi=pmi2
./your_mpi_application).
Tip
If you want to run an MPI job written in Python (for example, using mpi4py) you would simply need a
command like srun python script.py.
Job efficiency
Tip
You can check the real-time memory and cpu efficiency of your jobs directly in this webpage.
It is in your own and in all users' interest that all jobs running within Slurm are as efficient as possible, as this means that more science can be done with the limited resources available. You should take into account three different efficiencies: time, CPU and memory efficiency. For those "burro"s with GPUs, you should also consider GPU efficiency. We will include information about this shortly.
Time efficiency. This is the ratio of the actual wallclock time your job took until completion to the requested time. So, for example, if you set a time limit of one day for a job that only takes twelve hours to complete, the time efficiency of this job will be 50%.
CPU efficiency. This represents how efficiently allocated CPUs were being used while a job was running. For example, a job that was allocated two CPUs runs for ten hours, having a 100% load in both CPUs for the first five hours but one of the CPUs being completely idle for the remaining five hours, will have a CPU efficiency of 75%.
Memory efficiency. This represents the ratio of the peak memory consumption of your job to the allocated memory. Thus, for example, if a job's RAM consumption peaks at some point of the execution at 45GB when the allocated memory was 120GB, its Memory efficiency will be 37.5%.
Measuring your job's efficiencies
In order to help you make your jobs more efficient, you can use two tools that we have deployed in the "burros".
promseff for a single running job
After your job has finished (either succesfully completed, cancelled or timed-out), an efficiency report will be attached to the standard output of your job.
You can get the same report while the job is still running using the
promseff utility.
[user@burro:~]$ promseff -j 132503
#################~ JOB EFFICIENCY REPORT ~##################
# JobID: 132503 #
# Cluster: burro #
# Cores: 16 #
# Wall-clock time: 17.86 hours #
# User-CPU Utilized: 170.77 CPU-hours #
# User-CPU Efficiency: 59.74 % #
# System-CPU Utilized: 1.56 CPU-hours #
# System-CPU Load: 0.54 % #
# Memory Utilized: 7.06 GB #
# Memory Efficiency: 2.30 % #
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#
# If your job has low CPU Efficiency or you have doubts #
# about setting up a job, do not hesitate and contact us: #
# res_support@iac.es #
############################################################
Note
Due to restriction to access network-shared resources,
the reports will not be generated if
the output of the job is in the /home or /net. Please use the local
disks for the jobs.
rseff for multiple completed jobs
In order to help you make your jobs more efficient, you can use the tool
rseff installed in the burros (the original name of this tool is
reportseff as found
here). Check all the
options with the command rseff --help, but below we show an example
where we want to collect the efficiencies of the jobs that run in the
last day. We can see that some jobs had good CPU efficiency (> 80%), not
too bad memory efficiency (~30-50%), but terrible time efficiency in all
cases. When run in the terminal, the efficiencies will be colour coded
for easily spotting (in red), those efficiencies that you should strive
to improve for future jobs.
[user@burro:~]$ rseff --since d=1 -s CD
JobID State Elapsed TimeEff CPUEff MemEff
128652 COMPLETED 02:58:02 19.8% 60.0% 142.4%
128655 COMPLETED 03:14:01 1.0% 1.4% 28.2%
128656 COMPLETED 00:20:33 0.1% 2.3% 55.2%
128659 COMPLETED 00:01:29 0.2% 80.2% 31.2%
128661 COMPLETED 00:01:30 0.2% 83.8% 31.2%
128662 COMPLETED 00:09:30 5.3% 7.2% 109.0%
128663 COMPLETED 00:07:21 4.1% 7.7% 108.7%
Note
In the above rseff command example, we are getting the efficiencies only
for COMPLETED jobs (-s CD). This is due to a limitation of Slurm that
can only realiably measure job statistics for jobs that have finished with a
COMPLETED state. If your job reached a TIMEOUT or it was CANCELLED,
the statistics can be woefully wrong. Use the promseff utility described
above or check your job output for a correct efficiency report in those
cases.
Slurm information in all Burros
Commands like sinfo, squeue, etc. give, by default, information about
the Burro where you are running the command, but from any burro you can also get
information about another burro (or about all of them). For example:
[user@burro ~]$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
batch* up 7-00:00:00 1 mix burro
interactive up 8:00:00 1 mix burro
[user@burro ~]$ sinfo -M burro2
CLUSTER: burro2
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
batch* up 7-00:00:00 1 alloc burro2
batch_vip up 7-00:00:00 1 alloc burro2
interactive up 8:00:00 1 alloc burro2
[user@burro ~]$ squeue -M burro3,burro4,burro5
CLUSTER: burro3
JOBID PARTITION QOS NAME USER ST TIME TIME_LIMIT CPUS TRES_PER_NODE NODELIST(REASON)
CLUSTER: burro5
JOBID PARTITION QOS NAME USER ST TIME TIME_LIMIT CPUS TRES_PER_NODE NODELIST(REASON)
CLUSTER: burro4
JOBID PARTITION QOS NAME USER ST TIME TIME_LIMIT CPUS TRES_PER_NODE NODELIST(REASON)
140458 batch normal parallelcsv user2 PD 0:00 16:00:00 20 N/A (Resources)
140456 batch normal B1UNCl8W034 user3 R 11:40:25 2-00:00:00 10 N/A burro4
[user@burro ~]$ squeue -M all | head -n 20
CLUSTER: burro5
JOBID PARTITION QOS NAME USER ST TIME TIME_LIMIT CPUS TRES_PER_NODE NODELIST(REASON)
130333 interacti vip_dal interactive user4 R 2:48:09 8:00:00 2 N/A burro5
130332 interacti vip_dal interactive user4 R 2:48:17 8:00:00 2 N/A burro5
CLUSTER: burro6
JOBID PARTITION QOS NAME USER ST TIME TIME_LIMIT CPUS TRES_PER_NODE NODELIST(REASON)
[...]
ssummary command
Being able to quickly find which burros have cores available can be quite
convenient, so we make available the ssummary command (only in burros
running Slurm), which will tell you (for each public and open-project burro) the
total number of cores and how many are currently allocated and how many are free
(idle). For example:
[user@burro ~]$ ssummary
CLUSTER ALLOC IDLE OTHER TOTAL
------- ----- ---- ----- -----
burro5 4 44 0 48
burro6 0 48 0 48
burro7 0 256 0 256
burro 0 20 0 20
burro8 0 224 0 224
burro9 0 192 0 192
burro10 6 90 0 96
burro3 0 64 0 64
burro4 10 10 0 20
burro2 96 0 0 96
burro11 8 12 0 20