Run GPU Job
Request GPU resources, inspect available hardware and use GPU-related environment variables in HPC jobs.
Unreviewed migration
GPU resources
The table below summarises the CSG-managed resources for GPU:
| Resources | PATH/WHERE | Remark |
|---|---|---|
| CUDA from OS | /usr/local/cuda/ | |
| CUDA Containers | /share/apps/sif/x11vnc | See Software and Containers — Apptainer. |
| CUDA Module | module avail cuda/12.5.1-gcc-12.2.0-tk2uq2c | See Software and Containers — Environment Modules. |
| Conda/python Environments | /share/apps/noarch/miniforge3/envs/ | See python and conda. |
Submit GPU job
The job scheduler allocates GPU resources on a semi best-effort basis. If it cannot allocate the requested number of GPUs (usually due to minor hardware failures), it will still dispatch the job to run. This approach helps to avoid wasting the user's queued time. It is up to the user or job to handle this at the application level (such as exit anyway, or switch to use CPU).
GPU resources are tightly managed by the job scheduler. To use a GPU, users must specify the request in their job submission. A typical qsub command looks like:
qsub -b y -N gpu_job -pe smp 8 -l ngpus=2,gpu_model=A2,mem=16G,jobfs=10G GPU_JOB_CMDThe following GPU-related attributes are available:
| Attribute | Description |
|---|---|
ngpus | Integer value, the number of GPUs |
gpu_model | String value, specifying a particular GPU model |
gpu_code | String value, specifying a particular GPU family via its codename |
View available hardware
# This gives the details of the specified compute node. Remove the '-h' will print out
# the details of ALL compute nodes.
$ qhost -F ngpus,gpu_model,gpu_code -h wp-zeta-c20
HOSTNAME ARCH NCPU NSOC NCOR NTHR LOAD MEMTOT MEMUSE SWAPTO SWAPUS
----------------------------------------------------------------------------------------------
global - - - - - - - - - -
wp-zeta-c20.cse.unsw.edu.au lx-amd64 96 2 48 96 4.78 1007.4G 3.7G 977.0M 9.0M
Host Resource(s): hc:ngpus=2.000000
hf:gpu_code=GH100
hf:gpu_model=H100_NVL
# To view a GPU model summary:
$ qhost -F | grep gpu_model | sort -u
hf:gpu_model=A2
hf:gpu_model=H100_NVL
hf:gpu_model=L4
hf:gpu_model=L40S
hf:gpu_model=T1000_8GBJob environment
GPU resources are allocated and assigned on a per-job request basis. The following environment variables, containing details of the allocated resources, are injected into the job. It is up to the user/job to utilise this information as needed by the application.
| Variable | Description | Note/Example |
|---|---|---|
CUDA_DEVICE_ORDER | CUDA variable, fixed value, normally PCI_BUS_ID | |
CUDA_VISIBLE_DEVICES | CUDA variable, a list of comma-separated UUIDs | CUDA_VISIBLE_DEVICES: GPU-eaf4802f-e78d-e9d8-6381-afe98b201dae,GPU-aa3ba611-cc7d-c6db-bb32-905d732b4427 |
APPTAINERENV_CUDA_VISIBLE_DEVICES | Used by Apptainer container, value same as CUDA_VISIBLE_DEVICES | See Apptainer GPU docs. |