Please note that all the SIEpedia's articles address specific issues or questions raised by IAC users, so they do not attempt to be rigorous or exhaustive, and may or may not be useful or applicable in different or more general contexts.

Submit script files

Introduction

SLURM is the utility used at LaPalma for batch processing support, so all jobs must be run through it. This document provides information for getting started with job execution at LaPalma. Here we will describe the most important options and some examples of submission script files are available, but we recommend you check the SLURM Quick Start User Guide.

In order to keep the login nodes in a proper load, a 10 minutes limitation in the cpu time is set for processes running interactively in these nodes. Any execution taking more than this limit should be carried out through the queue system (see this FAQ).

Queues (QOS)

The user's limits are assigned automatically to each particular user (depending on the resources granted by the Access Committee). Anyway you are allowed to use the special queue debug in order to perform some fast short tests.

The specific limits assigned to each user depends on the priority granted by the access committee. Users granted with high priority hours will have access to a maximum of 2400 CPUs and a maximum wall clock limit of 72 hours. For users with low priority hours the limits are 1200 CPUs and 24 hours. If you need to increase these limits please contact the support group.

• class_a, class_b and class_c: Queues assigned by the access committee and where normal jobs will be executed, no special directive is needed to use these queues, they will be automatically assigned.
• debug: This queue is reserved for testing the applications before submitting them to the production queues. Only one job per user is allowed to run simultaneously in this queue, and the execution time will be limited to 30 minutes. The maximum number of cpus per application is 64. Only a limited number of jobs may be running at the same time in this queue. To use this queue add a directive in your script file, or also specify the queue when submitting without changing the script:

   #SBATCH --qos=debug
    - or -
   [lapalma1]$ sbatch --qos=debug script.sub

• interactive: Jobs submitted to this queue will run in the interactive (login) node. It is intended to run GUI applications that may exceed the interactive cpu time limit. Note that only sequential jobs are allowed. To use this queue launch the following command from login1 (see this FAQ):

   [lapalma1]$ salloc -p interactive

Submission directives

A job must contain a series of directives to inform the batch system about the characteristics of the job. These directives appear as comments in the job script, with the following syntax:

  #SBATCH --directive=<value>

Some common directives have a shorter version, you can use both forms:

  #SBATCH -d <value> 

Additionally, the job script may contain a set of commands to execute. If not, an external script must be provided with the 'executable' directive. Here you may find the most common directives (complete list here):

• -J ...: Name of the job
• --qos <queue_name>: The queue where the job is to be submitted. Let this field empty unless you need to use debug queue
• -t ...: walltime (use format hh:mm:ss or days-hh:mm:ss)
• -n ...: Number of tasks, this is the normal way to specify how many cores you want to use
• -o /path/to/file_out: Redirect standard output (stdout) to file_out (use /dev/null to ignore this output)
• -e /path/to/file_err: Redirect error output (stderr) to file_err (use /dev/null to ignore this output)
• -D <directory>: Execution will be performed in the specified directory (if it is not set, current directory will be used)

How to specify the submission options

You can specify these options in the command line:

   [lapalma1]$ sbatch -J <job_name> -t <days-HH:MM:SS> <your_executable>

But we highly recommend you write all the commands in a file (called submission script file) so you can reuse it when needed. That file should have next sections:

1. Submission file must be a executable script (although no execute permit is needed) beginning with line #!/bin/bash
2. SLURM options (as many as needed): #SBATCH -directive [<value>]
3. Modules to be loaded (as many as needed). Your environment variables will be stored when submitting your job and then used when executing the program. This could be a problem if your environment at submission time is not the proper one to execute your programs (for instance, no path to executables or dynamic libraries are set), so we recommend you begin cleaning your environment with module purge and then load only the modules required by your program.
4. Shell commands needed to run your application.

Once your script file is ready, you only need to use next command to submit it to the queue:

   [lapalma1]$ sbatch script_file

If you need more information about how to manage your jobs, check also the Useful Commands (executions) and the FAQs.

Environment variables

Although is not needed in most situations, there are also some SLURM environment variables you can use in your scripts if you need them

Examples of submission script files

Here you will find some examples about script files for different situations.

Note: If you copy and paste these examples, be careful because some unwanted spaces may be added at the beginning of each line: make sure that lines that contain parameters begin with #SBATCH and there are no spaces before these symbols.

Basic example (MPI)

You want to run your MPI program called myprogram_mpi using 64 cores (4 nodes) and that should take about 5 hours (add always some extra time because your application will be killed if it overpasses this wall time limit):

  #!/bin/bash
  #############################
  #SBATCH -J test_mpi
  #SBATCH -n 64
  #SBATCH -t 05:30:00
  #SBATCH -o %x-%j.out
  #SBATCH -e %x-%j.err
  #SBATCH -D .
  #############################

  module purge
  module load gnu openmpi/gnu

  mpirun ./myprogram_mpi

  # Use these other options if your MPI program does not run properly
  # srun  ./myprogram_mpi
  # srun --mpi=pmi2 ./myprogram_mpi

Comments:

• -n 64: this script will run the application myprogram_mpi on 64 cores (4 nodes)
• -D . The working directory will be the current one (where the submission was performed from)
• -o and -e: Two output files will be created, one for the standard output (-o, extension .out) and another one for errors (-e, extension .err). Note that we have used %x, so those files will be named using the job name specified with -J (test_mpi). We have also included the parameter %j in the names of those files, so the job ID will be added to them, in order to avoid overwriting the output files if we execute several times this script, since each execution will have a different job ID (this ID is shown when you submit the script using sbatch. You can also get it using squeue once the submission is done and the job has not finished yet). For instance, if your job name was test_mpi and the job ID was 1234, files will be named test_mpi-1234.out and test_mpi-1234.err.
• Remember that you cannot directly run MPI programs, you need to use srun or mpirun to execute them. If no more arguments are added, the number of slots specified by the SBATCH parameters will be used, but you can also force value using srun -n 20, srun -n $SLURM_NTASKS, etc. If you have problems running MPI programs (they do not initialize, or they are executed sequentially, change the command or options to run your program, you can use one of the next ones: mpirun, srun, srun --mpi=pmi2, etc.
• Do not forget to load all needed modules. For instance, if you want to execute VASP, you will need to use next commands and later use mpirun to run VASP:

   module purge 
   module load intel mkl vasp 

Basic example (OpenMP)

You want to run your OpenMP program called myprogram_omp (written in C or fortran) using 16 slots (this is the maximum number of available slots with shared memory to run OpenMP). This program should take about 30 minutes (add always some extra time because your application will be killed if it overpasses this wall time limit)

  #!/bin/bash
  #############################
  #SBATCH -J test_omp
  #SBATCH -n 16
  #SBATCH -t 00:45:00
  #SBATCH -o %x-%j.out
  #SBATCH -e %x-%j.err
  #SBATCH -D .
  #############################

  module purge
  module load gnu

  export OMP_NUM_THREADS=16
  ./myprogram_omp

Comments:

• CAUTION: Be sure you execute your OpenMP programs directly, and you do NOT use mpirun or srun (unless you have a hybrid MPI-OpenMP), since using them several instance of your OpenMP program will be repeated.
• Using this script our application myprogram_omp will be executed with 16 slots in one node (setting OMP_NUM_THREADS to 16 is not really needed, since by default the number of tasks will be used. If for any reason you want to execute with a different number of threads, then you can use this variable to set it).
• The working directory will be the current one (where the submission was performed from).
• -o and -e: Two output files will be created, one for the standard output (-o, extension .out) and another one for errors (-e, extension .err). Note that we have used %x, so those files will be named using the job name specified with -J (test_omp). We have also included the parameter %j in the names of those files, so the job ID will be added to them, in order to avoid overwriting the output files if we execute several times this script, since each execution will have a different job ID (this ID is shown when you submit the script using sbatch. You can also get it using squeue once the submission is done and the job has not finished yet). For instance, if your job name was test_omp and the job ID was 1234, files will be named test_omp-1234.out and test_omp-1234.err.

Jobs array

Jobs array and task generation can be used to run applications over different inputs like you were able to do with GREASY in past version of LaPalma.

Please, choose the proper script depending on your programs (parallel or sequential).

Example of Job Array (parallel programs)

  #!/bin/bash
  ##########################################################
  #SBATCH -J test_MPI_jobsarray
  #SBATCH -n 32
  #SBATCH -t 0-1:10:00
  #SBATCH --array=0-18:2
  #SBATCH -o test_jobsarray-%A-%j-%a.out
  #SBATCH -e test_jobsarray-%A-%j-%a.err
  #SBATCH -D .
  ##########################################################

  module purge
  module load gnu openmpi/gnu

  echo "#1 EXECUTING TASK ID: $SLURM_ARRAY_TASK_ID"
  fmtID=$(printf "%03d" $SLURM_ARRAY_TASK_ID)
  srun ./mpi_program -i input$fmtID.dat

  Submitted batch job 416

  test_jobsarray-416-416-0.err   #(%A = $SLURM_ARRAY_JOB_ID = 416, %j = $SLURM_JOBID = 416, %a = $SLURM_ARRAY_TASK_ID = 0)
  test_jobsarray-416-416-0.out
  test_jobsarray-416-417-2.err   #(%A = $SLURM_ARRAY_JOB_ID = 416, %j = $SLURM_JOBID = 417, %a = $SLURM_ARRAY_TASK_ID = 2)
  test_jobsarray-416-417-2.out
  test_jobsarray-416-418-4.err   #(%A = $SLURM_ARRAY_JOB_ID = 416, %j = $SLURM_JOBID = 418, %a = $SLURM_ARRAY_TASK_ID = 4)
  test_jobsarray-416-418-4.out
  test_jobsarray-416-419-6.err   #(%A = $SLURM_ARRAY_JOB_ID = 416, %j = $SLURM_JOBID = 419, %a = $SLURM_ARRAY_TASK_ID = 6)
  test_jobsarray-416-419-6.out
  test_jobsarray-416-420-8.err   #(%A = $SLURM_ARRAY_JOB_ID = 416, %j = $SLURM_JOBID = 420, %a = $SLURM_ARRAY_TASK_ID = 8)
  test_arrayjobs-416-420-8.out
  test_arrayjobs-416-421-10.err  #(%A = $SLURM_ARRAY_JOB_ID = 416, %j = $SLURM_JOBID = 421, %a = $SLURM_ARRAY_TASK_ID = 10)
  test_jobsarray-416-421-10.out
  test_jobsarray-416-422-12.err  #(%A = $SLURM_ARRAY_JOB_ID = 416, %j = $SLURM_JOBID = 422, %a = $SLURM_ARRAY_TASK_ID = 12)
  test_jobsarray-416-422-12.out
  test_jobsarray-416-423-14.err  #(%A = $SLURM_ARRAY_JOB_ID = 416, %j = $SLURM_JOBID = 423, %a = $SLURM_ARRAY_TASK_ID = 14)
  test_jobsarray-416-423-14.out
  test_jobsarray-416-424-16.err  #(%A = $SLURM_ARRAY_JOB_ID = 416, %j = $SLURM_JOBID = 424, %a = $SLURM_ARRAY_TASK_ID = 16)
  test_jobsarray-416-424-16.out
  test_jobsarray-416-425-18.err  #(%A = $SLURM_ARRAY_JOB_ID = 416, %j = $SLURM_JOBID = 425, %a = $SLURM_ARRAY_TASK_ID = 18)
  test_jobsarray-416-425-18.out

  [lapalma1]$ scancel 416
  [lapalma1]$ scancel 416_2
  [lapalma1]$ scancel 416_[6-8]
  [lapalma1]$ scancel 416_8 416_16

Example of Jobs Array (sequential programs)

 #!/bin/bash 
 ##############################################################################
 #SBATCH -J test_SEQ_jobsarray
 #SBATCH -n 16
 #SBATCH -t 0-0:30:00
 #SBATCH -o test_jobsarray_seq-%A-%j-%a.out
 #SBATCH -e test_jobsarray_seq-%A-%j-%a.err
 #SBATCH -D .
 #SBATCH --array=0-3:1
 ##########################################################

  module purge
  module load gnu

 # Specify how many executions will be performed in each node
 ''# Use N=16 to run an execution per core
 N=16

 for ((i=$SLURM_ARRAY_TASK_ID*$N+1; i <= $SLURM_ARRAY_TASK_ID*$N+$N; i++))
 do
   # Run your program. Make sure you use the & symbol to run your executions
   # in background, distributing them among the cores.
   # Use $i to specify the params and arguments
   ./my_seq_program -var=$i -file=input$i.dat &

   # Wait for completion after last iteration ($i % $N == 0)
   if ! expr $i % $N > /dev/null
   then
     wait
   fi
 done

Main

• Home Page
• WikiSandbox

Tutorials

HOWTOs

Users Tips

Main sites

• SIE Homepage
• IAC Homepage

Editing Help

• Cookbook
• PmWiki.BasicEditing

About the SIEpedia Accessibility

edit SideBar