Analysis / Job Submission

Request resources and submit batch jobs to the CSE HPC Gridengine scheduler with qsub.

The backend job scheduler is Gridengine. It functions similarly to PBS Pro / OpenPBS.

Resource requests

The table below summarises the major resource attributes which are commonly used in most jobs. There are also other attributes which are very helpful for fine-tuning how a job should be scheduled.

Resources	Attribute	Description	Default Value
Parallel Environments (`-pe`)	`smp`	Allocate X number of CPUs on the SAME compute node	Optional. If not specified, a job defaults to use 1 CPU.
Parallel Environments (`-pe`)	`mpi`	Allocate X number of CPUs from multiple compute nodes. This is mainly used by jobs that implement the Open MPI framework.
Resource request list (`-l`)	`mem`	The amount of memory limit that can be used by a job	Default 1024M
Resource request list (`-l`)	`jobfs`	The amount of disk space limit that can be used by a job	Default 1G
Resource request list (`-l`)	`walltime`	The run time limit (elapsed time) before a job gets killed by the job scheduler	Default 0:30:0
Resource request list (`-l`)	`ngpus`	The number of GPGPUs	N/A
project_name (`-P`)	`project_name`	Request a job to consume resource quota defined via Project. Check Project sub-section for details.	Optional. If not specified, default per-user quota is consumed instead.

Submit a batch job

A batch job can be submitted by using the qsub command, in the following pattern:

# submit a job which calls a script (bash, shell, python scripts etc)
qsub -N JOB_NAME -pe smp NUMBER_OF_CPU -l ATTR1=VAL1,ATTR2=VAL2 SCRIPT

# submit a job which calls a BINARY (anything which are not script, such as sleep, dd etc)
qsub -N JOB_NAME -pe smp NUMBER_OF_CPU -l ATTR1=VAL1,ATTR2=VAL2 -b y BINARY

Examples

# a very big sleep job that needs 16 x CPUs, 2 x GPGPUs, 64GB memory, 10G disk space
qsub -b y -N generic_gpgpu -pe smp 16 -l ngpus=2,mem=65G,jobfs=10G sleep 1m

# a smaller sleep job that requires the specific A2 GPGPU...
qsub -b y -N t1000_gpgpu -pe smp 8 -l ngpus=2,gpu_model=A2,mem=16G,jobfs=10G sleep 1m

# a big job runs on multiple H100 nodes inside the same physical rack/cabinet F (rack awareness)
qsub -b y -N h100_gpgpu -pe mpi 256 -l ngpus=2,gpu_model=H100,rack=f,mem=128G,jobfs=100G sleep 1m

For larger and more complex analyses, the qsub submission script can be very useful. A submission script contains pre-populated qsub parameters, and can be reused, distributed and version-controlled easily. It looks like:

#!/bin/bash
#
# It prints the actual path of the job scratch directory.
#$ -pe smp 8
#$ -j y
#$ -e $JOB_ID_$JOB_NAME.out
#$ -o $JOB_ID_$JOB_NAME.out
#$ -cwd
#$ -N dd_smp
#$ -l mem=1G,jobfs=110G,tmpfree=150G,walltime=00:30:00
#$ -P project_name

echo "$HOSTNAME $TMPDIR $jobfs"

# about 107GB
dd if=/dev/zero of=$TMPDIR/dd.test bs=512M count=200

To submit:

z1234567@login01:~$ qsub sge_dd_smp.sh
Your job 31 ("dd_smp") has been submitted

Local scratch

Local scratch is temporary storage! All data inside will be deleted upon job completion. Make sure data is copied back to somewhere!

Each compute node has a dedicated local storage which acts as the "Tier 0" scratch storage. When a job starts, it is given a dedicated (but temporary) directory on the scratch storage and its path is assigned to the variable $TMPDIR. Inside the submission script, $TMPDIR can be utilised in the following pattern:

cp -r $HOME/PROJECT_DATA $TMPDIR/

TOOL_BINARY -input $TMPDIR/PROJECT_DATA -output $TMPDIR/OUTPUT_DATA

# MAKE SURE the output data is copy back to your persistent storage location
cp -r $TMPDIR/OUTPUT_DATA $HOME/

It is generally a good idea to utilise this "Tier 0" local scratch as it gives the best disk performance, compared to network shared storage (such as home directory).

Analysis / Job Submission

Resource requests

Submit a batch job

Examples

Submission script

Local scratch

On this page