Submit script files

Introduction

SLURM is the utility used at LaPalma for batch processing support, so all jobs must be run through it. This document provides information for getting started with job execution at LaPalma. Here we will describe the most important options and some examples of submission script files are available, but we recommend you check the SLURM Quick Start User Guide.

In order to keep the login nodes in a proper load, a 10 minutes limitation in the cpu time is set for processes running interactively in these nodes. Any execution taking more than this limit should be carried out through the queue system (see this FAQ).

Queues (QOS)

The user's limits are assigned automatically to each particular user (depending on the resources granted by the Access Committee). Anyway you are allowed to use the special queue debug in order to perform some fast short tests.

Queues	Max Nodes	Wall time limit
class_a	288	72 hours
class_b	70	48 hours
class_c	70	24 hours
debug	4	30 min
interactive	1	6 hours

The specific limits assigned to each user depends on the priority granted by the access committee. Users granted with high priority hours will have access to a maximum of 286(13824) nodes(cores) and a maximum wall clock limit of 72 hours. For users with low priority hours the limits are 70(3360) nodes(cores) and 24 hours. If you need to increase these limits please contact the support group.

• class_a, class_b and class_c: Queues assigned by the access committee and where normal jobs will be executed, no special directive is needed to use these queues, they will be automatically assigned.

• debug: This queue is reserved for testing the applications before submitting them to the production queues. Only one job per user is allowed to run simultaneously in this queue, and the execution time will be limited to 30 minutes. The maximum number of cpus per application is 192. Only a limited number of jobs may be running at the same time in this queue. To use this queue add a directive in your script file, or also specify the queue when submitting without changing the script:

  #SBATCH --qos=debug
   - or -
  [lapalma1]$ sbatch --qos=debug script.sub
  

• interactive: Jobs submitted to this queue will run in the interactive (login) node. It is intended to run GUI applications that may exceed the interactive cpu time limit. Note that only sequential jobs are allowed. To use this queue launch the following command from the login node:

  [lapalma1]$ salloc -p interactive
  

Submission directives

A job must contain a series of directives to inform the batch system about the characteristics of the job. These directives appear as comments in the job script, with the following syntax:

#SBATCH --directive=<value>

Some common directives have a shorter version, you can use both forms:

#SBATCH -d <value>

Additionally, the job script may contain a set of commands to execute. If not, an external script must be provided with the 'executable' directive. Here you may find the most common directives (complete list here):

• -J ...: Name of the job

• --qos <queue_name>: The queue where the job is to be submitted. Let this field empty unless you need to use debug queue

• -t ...: Walltime must be set using format HH:MM:SS or DD-HH:MM:SS to a value greater than the real execution time for your application, bear in mind that your job will be killed after the period you specified. Shorter limits are likely to reduce the waiting time in the queue. If you do not specify any time limit, the maximum available in your assigned queue will be used.

• -n ...: Number of tasks, this is the normal way to specify how many cores you want to use

• -o /path/to/file_out: Redirect standard output (stdout) to file_out (use /dev/null to ignore this output)

• -e /path/to/file_err: Redirect error output (stderr) to file_err (use /dev/null to ignore this output)

• -D <directory>: Execution will be performed in the specified directory (if it is not set, current directory will be used)

Attention

Before submitting large jobs, please, perform some short tests to make sure your program is running fine. When running your jobs, check outputs and logs from time to time, and cancel the job if application fails.

How to specify the submission options

You can specify these options in the command line:

[lapalma1]$ sbatch -J <job_name> -t <days-HH:MM:SS> <your_executable>

But we highly recommend you write all the commands in a file (called submission script file) so you can reuse it when needed. That file should have next sections:

1. Submission file must be a executable script (although no execute permit is needed) beginning with line #!/bin/bash

2. SLURM options (as many as needed): #SBATCH -directive [<value>]

3. Modules to be loaded (as many as needed). Your environment variables will be stored when submitting your job and then used when executing the program. This could be a problem if your environment at submission time is not the proper one to execute your programs (for instance, no path to executables or dynamic libraries are set), so we recommend you begin cleaning your environment with module purge and then load only the modules required by your program.

4. Shell commands needed to run your application.

Once your script file is ready, you only need to use next command to submit it to the queue:

[lapalma1]$ sbatch script_file

If you need more information about how to manage your jobs, check also the Useful Commands (executions) and the FAQs.

Environment variables

Although is not needed in most situations, there are also some SLURM environment variables you can use in your scripts if you need them.

Variable	Meaning
SLURM_JOBID	Specifies the job ID of the executing job
SLURM_NPROCS	Specifies the total number of processes in the job
SLURM_NNODES	Is the actual number of nodes assigned to run your job
SLURM_PROCID	Specifies the MPI rank (or relative process ID) for the current process. The range is from 0-(SLURM_NPROCS-1)
SLURM_NODEID	Specifies relative node ID of the current job. The range is from 0-(SLURM_NNODES-1)
SLURM_LOCALID	Specifies the node-local task ID for the process within a job
SLURM_NODELIST	Specifies the list of nodes on which the job is actually running
SLURM_ARRAY_TASK_ID	Task ID inside the job array
SLURM_ARRAY_JOB_ID	Job ID (it will be the same for all jobs array, the same as the SLURM_JOBID of the first task).