The queue system used for the public and open-project "burros" is
`Slurm `__, widely used in many research
institutes and supercomputing centres. You can find a general guide on
how to use Slurm at IAC machines `here <../../software/slurm.html>`__.
Below we describe the default configuration and some commands specific
to the Slurm installation in the IAC **public** "burros". **Open-project**
"burros" share most of the configuration of a **public** "burro", but with some
usage conditions, which are described in the section on
`open-project burros <#usage-conditions-in-open-project-burros>`_ below.
Default Slurm configuration in a public "burro"
===============================================
.. note::
This configuration can be modified in project "burros" to take
into account the needs of each research group.
When a workstation is shared amongst several users it is easy to
oversubscribe it (e.g. to run more processes than available cores),
which makes all applications to perform worse and it is usually
detrimental to all users. To avoid this problem, the default Slurm
configuration in a "burro" is as follows:
Limited CPU power
~~~~~~~~~~~~~~~~~
**A default ssh or graphical session is limited to the CPU power equivalent to 3/4
of a core**. This should be enough for editing files, checking on the
state of the machine, etc. but certainly not enough for running any
intensive computer program. For this, you will need to use the Slurm
queueing system, either using an interactive session or submitting a
batch job.
.. important::
Slurm accounts are independent of the Linux system accounts, so
before you use Slurm you will need to get a Slurm account:
- if using a project burro, contact the PI of the project.
- if using a public or an open-project burro, you can activate an account by
yourself with the ``slurm_addme`` command. The following
example shows how a first call to the ``sinter`` command
(see section *Getting and interactive session* below) is not
successful because the account for ``user`` was not yet
created. ``slurm_addme`` creates the account and the
following ``sinter`` command is successful.
::
[user@burro ~]$ sinter -n 2
salloc: error: Job submit/allocate failed: Invalid account or account/partition combination specified
[user@burro ~]$ slurm_addme
Associations =
U = user A = ddgroup C = burro
Non Default Settings
[user@burro ~]$ sinter -n 2
salloc: Granted job allocation 127003
salloc: Nodes burro are ready for job
(sinter) [user@burro ~]$
Partitions configuration
~~~~~~~~~~~~~~~~~~~~~~~~
In order to have a reasonable balance of batch/interactive jobs, **we
configure two "partitions" (or "queues")**.
- ``batch``, meant for **long heavy-computation jobs**, with the
following limits:
- Max CPUs: number of hardware threads of node (*if hardware
threads are disabled in the node, this is the same as the
number of cores*)
- Max time: 2 days (**QOS**: ``normal``) or 7 days (**QOS**: ``long``). With Slurm
you can specify a Quality of Service (**QOS**) for each job. We have
configured the ``batch`` partition with two QOSs: ``normal`` (default
QOS), with max time of jobs of 2 days; ``long``, with max time of jobs
of 7 days, but with a greatly reduced priority when compared to the
``normal`` QOS. If possible, try to avoid using the ``long`` QOS (it is
much better if you can cut your job in smaller parts or if you can prepare
your application to frequently create checkpoints that can be used to
restart your application with the state saved by a previous run). If
really necessary, request this QOS with ``--qos=long``.
- ``interactive``, meant for **lighter interactive jobs**, with the
following limits:
- Max CPUs per job: number of hardware threads of node
- Max CPUs per partition: 2 \* (number of hardware threads of
node) [via oversubscription]
- Max time: 8 hours
At any time you can check the actual configuration of the partitions
with the ``sinfo`` command. For example:
::
$ sinfo -O SocketCoreThread,Partition,Maxcpuspernode,time
S:C:T PARTITION MAX_CPUS_PER_NODE TIMELIMIT
2:28:2 batch* 112 7-00:00:00
2:28:2 interactive 112 8:00:00
.. note::
While having fine-grained control of where each process runs
is not necessary for many jobs, those running parallel applications and
advanced users might want to read our notes in section `CPU Management
<../../software/slurm.html#cpu-management>`__).
.. important::
**It might seem obvious, but remember that the "interactive" queue is for
"interactive" jobs only!**
**Acceptable** use cases for the interactive queue are, for example: if you
need to test whether your code compiles OK with different compilation
options; if you are writing code and need to test if it works as you
develop it; if you are debugging code; if you need to run a GUI
application, etc.
**Non-acceptable** use cases are running long CPU-intensive jobs that do
not require your interaction in a long time (10 minutes or more).
This is so because jobs in these two partitions can overlap, so ``batch``
jobs can run alongside ``interactive`` ones (and this assumes that
interactive jobs are not as computation intensive as batch ones, and
allows a controlled oversubscription of the workstation).
In total, up to three times the number of hardware threads could be
allocated at the same time (one in the ``batch`` partition, and two in the
``interactive`` partition). This works OK *only* if the jobs in the
``interactive`` partition are light jobs, GUI apps interactions, Python
interactive sessions, etc.
**Don't abuse the system by submitting long CPU-intensive jobs in the
"interactive" queue or the system administrators will show you a yellow
(or even a red) card!**
Tracking Memory usage
~~~~~~~~~~~~~~~~~~~~~
In a shared "burro", **memory usage also needs to be tracked**, so
that we ensure that a job cannot saturate the available RAM (which
can negatively impact on other jobs' performance and even bring the
whole system down). The way **we track memory usage is as follows**:
.. note::
Memory limitation is activated by default in all public and open-project "burro"s,
but not in project ones. If you want your "burro" to have this
limitation, please let us know.
- by default, when submitting a job to Slurm, **each allocated CPU
will also be allocated a default RAM amount**. Since we have to
guarantee that a machine at full capacity does not consume more
memory than available, and given that up to three times the number
of hardware threads could be allocated at any given time (see
above), the default RAM allocated per CPU is:
.. math:: (1/3) Total\_memory / N_\mathrm{threads}
As an example, one of our "burro"s has 32 hardware threads, and a
total of 256 GB of RAM, so the default RAM allocated per CPU is
2.67 GB, which can be checked with the following command (output
is in MB):
::
$ scontrol show config | grep DefMemPerCPU
DefMemPerCPU = 2667
- if more/less memory is required, you can use the ``--mem-per-cpu``
or the ``--mem`` options to specify the required amount.
For example, if we only need one CPU, but we require 60 GB of
memory, the default value of 2.67 GB will not be enough, and the
job will be killed when this value is exceeded. To avoid that, we
can simply submit a job with a batch script similar to the one
below (see section *Batch jobs* below for more information on
batch scripts).
::
#!/bin/bash
#############################
#SBATCH -J test
#SBATCH -n 1
#SBATCH -t 00:15:00
#SBATCH --mem=60G
#############################
./your_job
Sometimes, it is perhaps easier to specify the amount of memory
requested per process (for example, in MPI parallel applications),
so in that case we could use a batch script similar to:
::
#!/bin/bash
#############################
#SBATCH -J test
#SBATCH -n 10
#SBATCH -t 00:15:00
#SBATCH --mem-per-cpu=6G
#############################
srun ./your_mpi_job
Tracking GPU usage
~~~~~~~~~~~~~~~~~~
In those **"burro"s that have GPUs, their usage is also tracked**
with Slurm, so that a user can have exclusive access to some or all
the GPUs. Slurm is configured in a way that jobs requesting a GPU
have a much larger priority than other jobs, but non-GPU jobs can
also use the "burro". In the `Introduction <./index.html>`__ section
you can see the table of currently available "burro"s, and whether
they have GPUs.
The way that GPU usage and tracking is integrated with Slurm in a
"burro" is as follows:
- **You will be able to use the GPU card only from inside a Slurm
job**, either interactive or batch. As a matter of fact, outside
of Slurm it will look like the "burro" has no GPUs (e.g. the
``nvidia-smi`` command will report a
``Failed to initialize NVML: Unknown Error``).
- **You request a GPU by adding the Slurm option** ``--gres=gpu``.
Once allocated, you will have exclusive access to the GPU for the
duration of the job.
You can get more information and examples of use of the GPUs at the
burros `here `__.
Fairshare
~~~~~~~~~
.. important::
The available resources in a "burro" are shared equally among all
users, and thanks to the feature called "FairShare", **Slurm will
give higher priority to those users that have used less than their
share**.
**Everytime you run a job, you are "billed" for all the resources you
get allocated (i.e. number of CPUs, RAM amount and GPUs)**, so it is
in your interest to request only the actual resources needed for your
job (regarding RAM, you should allow some safety buffer to avoid
exceeding the allocated memory and your job being killed).
The section *Job efficiency* below explains the three different job
efficiencies that you should care about and how to measure them. More
information on this can also be found in our `Sep'23
talk `__.
Integration of Slurm with periodic reboots
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Due to security considerations the IAC "burros" need to be rebooted regularly
whenever security patches for any of the Ubuntu packages are published. The
exact date of these reboots is not fixed, but we can expect in the order of one
or two reboots per month.
For machines without Slurm these reboots can be quite disruptive since the
reboots will kill all running processes (messages are broadcasted to all open
terminals in each "burro" to be rebooted, but it is quite easy to miss them if
you are executing long-running processes).
"Burros" using Slurm nicely integrate the information about periodic reboots so
as to make sure that no jobs will be running when the workstation is
rebooted. See the "note" below to understand how this feature works.
.. note::
Whenever a reboot is scheduled, we automatically create a Slurm reservation
of 30 minutes around the exact time when the reboot will take place. For
example, if a reboot is scheduled for tomorrow at 07:00, a Slurm reservation
of 30 minutes will be placed around that time and Slurm will make sure that
no jobs can be running during that reservation.
So, if you submit now a 48 hours job, it will not start running before the
reboot because if it were to use the full allocated 48 hours it would clash
with the reservation of tomorrow. In this case, you can either reduce the
allocation of the job so that it will end before the reservation begins, or
you can leave the job in the queue and it will start running after the reboot
whenever Slurm can find a suitable allocation slot for it.
You can see if there are planned reservations by using the command:
::
scontrol show res
Also, if you submit a job that clashes with a planned reservation,
you will see when using the ``squeue`` command that the job stays in the queue
in the ``PENDING`` (``PD``) state, and that the ``NODELIST (REASON)`` column reads
something like ``(ReqNodeNotAvail, Reserved for maintenance)``.
.. important::
As mentioned below, `access to network-shared resources
<#accessing-network-shared-resources>`_ can give troubles in certain cases
due to Kerberos tickets expiring. Integrated with Slurm we have a plugin
called `AUKS `__ which takes care of
renewing your Kerberos tickets for a certain time period, which can alleviate
most of the Kerberos+Slurm problems.
But **the AUKS plugin "magic" does not survive a reboot**, so if your jobs
make use of any network-shared resources (like ``/home`` or ``/net``) and
they are scheduled to run after a reboot, they will fail because they will
not have access to those resources. To avoid these problems, try to move your
data to the local disks of the burros.
Integration of Slurm (and HTCondor) with the CPD temperature sensor
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To protect the compute nodes (the *burros*) from overheating in case of an
air conditioning failure or any other incident in the Data Centre (CPD,
*Centro de Procesamiento de Datos*), the temperature in the CPD is
monitored continuously and fed into both of our batch systems.
A simple three-state model is used, based on two temperature thresholds
(currently 28°C and 30°C):
* **OK** — temperature at or below 28°C: everything operates normally.
* **SOFT** — temperature above 28°C: no new jobs are accepted. In Slurm,
the nodes are set to ``DRAIN`` with reason ``temperature_soft``, so any
currently running jobs are allowed to finish. In HTCondor, the ``startd``
daemons are stopped, which terminates running Condor jobs quickly.
* **HARD** — temperature above 30°C: in addition to the SOFT actions, any
Slurm jobs still running are also killed by setting the nodes ``DOWN``
with reason ``temperature_hard``.
By default these jobs are requeued and will start again on their own once the
temperature drops back to OK and the nodes are resumed. Up to 5 automatic
requeues per job are allowed; beyond that, the job is held with reason
``JobHoldMaxRequeue``.
If for any reason you would prefer that your job is not requeued in this case,
you can add ``#SBATCH --no-requeue`` to you submission script, and then it
will end up as ``NODE_FAIL`` and it should be resubmitted manually.
Interactive sessions (``sinter``) cannot be requeued by their nature — the
session is tied to your terminal, and you will see it terminate.
Once the temperature situation is over and the temperature is again ≤28ºC, then
both systems go back to normal operation and jobs will be accepted again.
State transitions due to temperature changes in the CPD trigger automatic
notifications in the **#computing/burros** Zulip channel.
.. _getting-an-interactive-session:
Getting an interactive session
==============================
Before the installation of
`Slurm `__, users
``ssh``-ed to the "burros" and executed jobs interactively in that
session. As stated above, this will now (by default) give you only the
CPU power equivalent to 3/4 of a core. In order to obtain a more
powerful interactive session you need to make use of the ``sinter``
command, specifying the number of cores requested and the time limit for
the session. The syntax for ``sinter`` is:
::
sinter -n -t
For example, if you want to request a 10 core interactive session for
two and a half hours, you could do (note that ``(sinter)`` is added to
your command line to remind you that you are now in a Slurm interactive
session):
::
[user@burro ~]$ sinter -n 10 -t 02:30:00
salloc: Granted job allocation 127221
salloc: Nodes burro are ready for job
(sinter) [user@burro ~]$
.. note::
- Do not ask for more cores than you need, as other people may be
waiting to use the "burro".
- After the requested time has passed, the session will expire
and ``sinter`` must be called again.
- The shorter the allocation time, the more likely that your job
can start right away.
- If you finish working, type ``exit`` to quit the interactive
session, so you can make the cores available for other users.
- For advanced users, note that ``sinter`` is just a wrapper
around the standard
`salloc `__ Slurm
command.
MPI Jobs
~~~~~~~~
If you want to test MPI jobs while in the interactive session, you just
need to use the ``srun`` command, while specifying the number of tasks
you want to use (less or equal to the number of cores you requested for
the interactive session). So, for example, after requesting the 10 core
interactive session as per the example above, you could simply run a 4
core MPI job by simply running:
::
(sinter) [user@burro ~]$ srun -n 4 ./your_mpi_application
.. note::
Not specifying the number of tasks to use and just executing ``srun
./your_mpi_application`` is equivalent to requesting all the cores requested
for the interactive session.
.. warning::
If your application has been compiled with Intel MPI, you should add
``--mpi=pmi2`` to your ``srun`` command (for example, ``srun -n 4 --mpi=pmi2
./your_mpi_application``).
.. tip::
Running a Python script using MPI would simple require ``srun python
script.py``.
Running `IPython `__ in parallel (using MPI) in an
interactive session is doable (see `IPython Parallel
`__), but certainly
not straightforward. If you just execute ``ipython`` in an interactive
session and then try to load MPI with ``mpi4py`` (either directly or via
another package that you use) you will receive an error message that says
``The application appears to have been direct launched using "srun", but OMPI
was not built with SLURM's PMI support and therefore cannot execute``. If you
really need this, please do get in touch with us and we will try to get it
working.
For debugging purposes, you can launch IPython in an interactive session with
``srun -n 1 --pty ipython``. This will give you only one MPI rank (the same
as if you were to launch IPython without Slurm), but your code will be able
to load the MPI libraries without errors and if it uses other
multi-processing libraries it will be able to use the number of cores
requested for the interactive session.
Batch jobs
==========
For general information about Slurm and how to submit batch jobs, please
refer to the `IAC-wide guide to Slurm <../../software/slurm.html>`__.
Below we provide some information applicable only to the "burros".
Accessing network shared resources
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
While accessing your data or scripts from the network (for example using your
``/home`` directory or the NAS) would be very convenient, there are a number of
technical difficulties integrating Kerberos with Slurm in a transparent way,
which means that some jobs will fail in a difficult to debug way (e.g. in some
cases the jobs will succeed while in others, when your Kerberos keys expire,
will fail).
.. important::
Due to the issues mentioned above, **accessing data or scripts directly from
the network is highly discouraged**. This includes files stored in ``/net``
and ``/home`` volumes (files stored in the BASTO storage system accessed via
``/storage`` do not pose this problem). To avoid these problems, try to move
your data to the local disks of the burros.
If you are not able to move your data to local disks or ``/storage`` then you
can rely on the AUKS plugin installed, which will take care of renewing your
Kerberos tickets for a certain time period, which can alleviate most of the
Kerberos+Slurm problems. To make sure that the plugin works right, make sure
that you have a valid Kerberos ticket and that you can access the AUKS
renewer service by running the following before submitting your job:
::
$ kinit
Password for @IAC.ES:
$ /opt/SIE/Slurm/extra/auks/bin/auks -p
Auks API request succeed
If you don't get "Auks API request succeed" or your job fails accessing the
network shared resources despite having run this, please do get in touch with
us.
Getting job notifications by e-mail
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you want to receive notification by e-mail of when your job begins
and ends, add the following to the "batch script" (see `Batch
jobs <../../software/slurm.html#batch-jobs>`__).
::
#SBATCH --mail-type=BEGIN,END
#SBATCH --mail-user=
.. warning::
Please note that mails will only be sent to IAC mail addresses.
As an example, when your job ends, you will receive an e-mail containing
information like the following:
::
######################## JOB EFFICIENCY REPORT ########################
# Job ID: 127267
# Cluster:
# User/Group: /
# State: COMPLETED (exit code 0)
# Cores: 1
# CPU Utilized: 00:00:11
# CPU Efficiency: 84.62% of 00:00:13 core-walltime
# Wall-clock time: 00:00:13
# Memory Utilized: 1.30 MB
#######################################################################
.. _mpi-jobs-1:
MPI Jobs
~~~~~~~~
Slurm and OpenMPI in the "burros" are nicely integrated, so when
submitting a batch MPI job you simply have to use the command ``srun``,
without worrying about the number of cores needed, which is taken
directly from the Slurm allocation, or the Process Management Interface
(PMI). As a basic example, your submission script could be simply as
follows:
::
#!/bin/bash
#############################
#SBATCH -J your_job
#SBATCH -n 4
#SBATCH -t 00:01:00
#SBATCH -o %j.out
#SBATCH -e %j.err
#SBATCH -D .
#############################
srun ./your_mpi_application
.. warning::
If your application has been compiled with Intel MPI, you should add
``--mpi=pmi2`` to your ``srun`` command (for example, ``srun -n 4 --mpi=pmi2
./your_mpi_application``).
.. tip::
If you want to run an MPI job written in Python (for example, using `mpi4py
`__) you would simply need a
command like ``srun python script.py``.
.. _sharing-your-allocation-with-another-terminal:
Sharing your allocation with another terminal
=============================================
Sometimes it can be useful to temporarily have access to the resources granted
to an allocation (be it an interactive session or a batch job) from another
terminal.
For example, when requesting an interactive session with GPU access, you only
have access to the allocated GPU from within this interactive session. Thus, if
you want to monitor the GPU usage from another terminal (for example with
commands ``nvidia-smi`` or ``nvtop``) you will not be able to do it unless you
request to "share" the allocated resources. In order to do that, in a new
terminal (without the need to run ``sinter`` or any other Slurm command) you can
issue a command like the following (with being the JOBID given by the
``squeue`` command for the allocation you want to share):
::
srun --pty --overlap --jobid /bin/bash
This will allow you to use and share the resources given to that job. When
finished, you can simply run ``exit``, and this terminal will stop sharing the
resources, but the original job will continue running unaltered.
Job efficiency
==============
.. tip::
You can check the real-time memory and cpu efficiency of your jobs
directly in `this webpage `__.
It is in your own and in all users' interest that all jobs running
within Slurm are as efficient as possible, as this means that more
science can be done with the limited resources available. You should
take into account three different efficiencies: time, CPU and memory
efficiency. *For those "burro"s with GPUs, you should also consider GPU
efficiency. We will include information about this shortly*.
- **Time efficiency**. This is the ratio of the actual wallclock time
your job took until completion to the requested time. So, for
example, if you set a time limit of one day for a job that only takes
twelve hours to complete, the time efficiency of this job will be
50%.
- **CPU efficiency**. This represents how efficiently allocated CPUs
were being used while a job was running. For example, a job that was
allocated two CPUs runs for ten hours, having a 100% load in both
CPUs for the first five hours but one of the CPUs being completely
idle for the remaining five hours, will have a CPU efficiency of 75%.
- **Memory efficiency**. This represents the ratio of the peak memory
consumption of your job to the allocated memory. Thus, for example,
if a job's RAM consumption peaks at some point of the execution at
45GB when the allocated memory was 120GB, its Memory efficiency will
be 37.5%.
Measuring your job's efficiencies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In order to help you make your jobs more efficient, you can use two
tools that we have deployed in the "burros".
``promseff`` for a single running job
-------------------------------------
After your job has finished (either succesfully completed, cancelled or
timed-out), an efficiency report will be attached to the standard output
of your job.
You can get the same report while the job is still running using the
``promseff`` utility.
::
[user@burro:~]$ promseff -j 132503
#################~ JOB EFFICIENCY REPORT ~##################
# JobID: 132503 #
# Cluster: burro #
# Cores: 16 #
# Wall-clock time: 17.86 hours #
# User-CPU Utilized: 170.77 CPU-hours #
# User-CPU Efficiency: 59.74 % #
# System-CPU Utilized: 1.56 CPU-hours #
# System-CPU Load: 0.54 % #
# Memory Utilized: 7.06 GB #
# Memory Efficiency: 2.30 % #
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#
# If your job has low CPU Efficiency or you have doubts #
# about setting up a job, do not hesitate and contact us: #
# res_support@iac.es #
############################################################
.. note::
Due to restriction to `access network-shared resources <#accessing-network-shared-resources>`_,
the reports will not be generated if
the output of the job is in the ``/home`` or ``/net``. Please use the local
disks for the jobs.
``rseff`` for multiple completed jobs
-------------------------------------
In order to help you make your jobs more efficient, you can use the tool
``rseff`` installed in the burros (the original name of this tool is
*reportseff* as found
`here `__). Check all the
options with the command ``rseff --help``, but below we show an example
where we want to collect the efficiencies of the jobs that run in the
last day. We can see that some jobs had good CPU efficiency (> 80%), not
too bad memory efficiency (~30-50%), but terrible time efficiency in all
cases. When run in the terminal, the efficiencies will be colour coded
for easily spotting (in red), those efficiencies that you should strive
to improve for future jobs.
::
[user@burro:~]$ rseff --since d=1 -s CD
JobID State Elapsed TimeEff CPUEff MemEff
128652 COMPLETED 02:58:02 19.8% 60.0% 142.4%
128655 COMPLETED 03:14:01 1.0% 1.4% 28.2%
128656 COMPLETED 00:20:33 0.1% 2.3% 55.2%
128659 COMPLETED 00:01:29 0.2% 80.2% 31.2%
128661 COMPLETED 00:01:30 0.2% 83.8% 31.2%
128662 COMPLETED 00:09:30 5.3% 7.2% 109.0%
128663 COMPLETED 00:07:21 4.1% 7.7% 108.7%
.. note::
In the above ``rseff`` command example, we are getting the efficiencies only
for *COMPLETED* jobs (``-s CD``). This is due to a limitation of Slurm that
can only realiably measure job statistics for jobs that have finished with a
*COMPLETED* state. If your job reached a *TIMEOUT* or it was *CANCELLED*,
the statistics can be woefully wrong. Use the ``promseff`` utility described
above or check your job output for a correct efficiency report in those
cases.
Slurm information in all Burros
===============================
Commands like ``sinfo``, ``squeue``, etc. give, by default, information about
the Burro where you are running the command, but from any burro you can also get
information about another burro (or about all of them). For example:
::
[user@burro ~]$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
batch* up 7-00:00:00 1 mix burro
interactive up 8:00:00 1 mix burro
[user@burro ~]$ sinfo -M burro2
CLUSTER: burro2
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
batch* up 7-00:00:00 1 alloc burro2
batch_vip up 7-00:00:00 1 alloc burro2
interactive up 8:00:00 1 alloc burro2
[user@burro ~]$ squeue -M burro3,burro4,burro5
CLUSTER: burro3
JOBID PARTITION QOS NAME USER ST TIME TIME_LIMIT CPUS TRES_PER_NODE NODELIST(REASON)
CLUSTER: burro5
JOBID PARTITION QOS NAME USER ST TIME TIME_LIMIT CPUS TRES_PER_NODE NODELIST(REASON)
CLUSTER: burro4
JOBID PARTITION QOS NAME USER ST TIME TIME_LIMIT CPUS TRES_PER_NODE NODELIST(REASON)
140458 batch normal parallelcsv user2 PD 0:00 16:00:00 20 N/A (Resources)
140456 batch normal B1UNCl8W034 user3 R 11:40:25 2-00:00:00 10 N/A burro4
[user@burro ~]$ squeue -M all | head -n 20
CLUSTER: burro5
JOBID PARTITION QOS NAME USER ST TIME TIME_LIMIT CPUS TRES_PER_NODE NODELIST(REASON)
130333 interacti vip_dal interactive user4 R 2:48:09 8:00:00 2 N/A burro5
130332 interacti vip_dal interactive user4 R 2:48:17 8:00:00 2 N/A burro5
CLUSTER: burro6
JOBID PARTITION QOS NAME USER ST TIME TIME_LIMIT CPUS TRES_PER_NODE NODELIST(REASON)
[...]
``ssummary`` command
~~~~~~~~~~~~~~~~~~~~
Being able to quickly find which burros have cores available can be quite
convenient, so we make available the ``ssummary`` command (only in burros
running Slurm), which will tell you (for each public and open-project burro) the
total number of cores and how many are currently allocated and how many are free
(idle). For example:
::
[user@burro ~]$ ssummary
CLUSTER ALLOC IDLE OTHER TOTAL
------- ----- ---- ----- -----
burro5 4 44 0 48
burro6 0 48 0 48
burro7 0 256 0 256
burro 0 20 0 20
burro8 0 224 0 224
burro9 0 192 0 192
burro10 6 90 0 96
burro3 0 64 0 64
burro4 10 10 0 20
burro2 96 0 0 96
burro11 8 12 0 20