Running Jobs on the New Slurm HPC (Alpha)

How to use modules, mamba environments and submit jobs on the new Debian 13 / Slurm cluster, with a resource-request cheat sheet for existing Gridengine users.

This covers the new Debian 13 / Slurm cluster (gxXX node names), currently in alpha. For the current production Gridengine system, see HPC User Guide and Analysis / Job Submission.

Slurm overview

GPU hardware available

Partition	Node(s)	GPU model	GRES string
`general`	`gx02`, `gx05`, `gx06`	NVIDIA L4	`gpu:nvidia_l4`
`general`	`gx11`	NVIDIA L40S	`gpu:nvidia_l40s`
`large`	`gx21`	NVIDIA H100 NVL	`gpu:nvidia_h100_nvl`
`large`	`gx30`, `gx31`	RTX PRO 6000 Blackwell Server Edition	`gpu:nvidia_rtx_pro_6000_blackwell_server_edition`

A partition can contain more than one GPU model, so always specify the exact gpu:<model> string in your --gres request rather than relying on the partition alone.

Resource requests: coming from Gridengine

If you already know qsub, the table below maps the resource attributes you're used to onto their Slurm equivalents on this cluster.

Need	Gridengine (`qsub`)	Slurm (`srun`/`sbatch`)	Notes
Job name	`-N NAME`	`--job-name=NAME`
CPUs, same node	`-pe smp N`	`--cpus-per-task=N`
CPUs, multiple nodes	`-pe mpi N`	`--ntasks=N` (with `--nodes=`)
Memory	`-l mem=X`	`--mem=X`	Per node.
Local scratch disk	`-l jobfs=X` / `tmpfree=X`	`--tmp=X`	`--tmp` is a scheduling constraint (ensures the node has enough local disk configured); it does not enforce a hard quota at runtime.
Local scratch path	`$TMPDIR`	`$TMPDIR` or `$SLURM_JOB_SCRATCH_DIR`	Both set automatically to `/srv/scratch/$SLURM_JOB_ID`, created per-job and removed on job completion.
Runtime limit	`-l walltime=HH:MM:SS`	`--time=HH:MM:SS`	Capped by QOS per partition: `general` max 10:00:00, `large` max 04:00:00.
GPU count/model	`-l ngpus=N,gpu_model=X`	`--gres=gpu:<model>:N`	See the GPU hardware table above.
Project/quota	`-P project_name`	`--account=NAME`	Available accounts: `general` (catch-all), `csg`, `binf_np`, `pag`, `cruise`, `dglab`.
Output/error files	`-o FILE -e FILE -j y`	`--output=FILE --error=FILE`
Working directory	`-cwd`	(default — Slurm already uses the submission directory)

Cluster/queue commands

Need	Gridengine	Slurm
Queue/node status	`qstat -f -u "*"`	`sinfo`
My jobs	`qstat -u $USER`	`squeue -u $USER` or `squeue --me`
Job detail	`qstat -j JOB_ID`	`scontrol show job JOB_ID`
Cancel a job	`qdel JOB_ID`	`scancel JOB_ID`
Accounting history	`qacct -j JOB_ID`	`sacct -j JOB_ID`

Modules for the new Debian 13 / zen4 stack live under /mnt/apps/Modules/. Set MODULEPATH to all three trees shown below — linux-debian13-x86_64 is required alongside linux-debian13-zen4 (some packages, e.g. mamba's dependencies, are registered under the generic x86_64 bucket):

source /etc/profile.d/lmod.sh
export MODULEPATH=/mnt/apps/Modules/noarch/:/mnt/apps/Modules/linux-debian13-x86_64/:/mnt/apps/Modules/linux-debian13-zen4/
module avail
module load <module-name>

Software: mamba / conda environments

A centrally-managed mamba build is available as a module (see above). It is built for the zen4 CPU microarchitecture, so it only runs on gx compute nodes — not on login nodes (vx*), which raises Illegal instruction if attempted there.

module load mamba-2.3.0-iiqe4tc
mamba env list
mamba create -p ~/mamba/<env-name> python=3.12

See the "Running a job" section below for how to activate and use an environment inside an actual job (interactive or batch) — the activation steps differ slightly from running mamba activate in an already-initialized personal shell.

Running a job

Interactive (`srun`)

srun --partition=large --gres=gpu:nvidia_rtx_pro_6000_blackwell_server_edition:1 --pty bash

source /etc/profile.d/lmod.sh
export MODULEPATH=/mnt/apps/Modules/noarch/:/mnt/apps/Modules/linux-debian13-x86_64/:/mnt/apps/Modules/linux-debian13-zen4/
module load mamba-2.3.0-iiqe4tc
eval "$(mamba shell hook -s bash)"
mamba activate ~/mamba/<env-name>

The eval "$(mamba shell hook -s bash)" line is required even here: a fresh srun session does not have mamba activate wired into its shell unless your own .bashrc already runs mamba shell init. Running it explicitly works regardless of your personal shell setup.

Batch (`sbatch`)

Example — a PyTorch job on an RTX PRO 6000:

pytorch_test.py:

import torch, time

print("CUDA available:", torch.cuda.is_available())
print("Device:", torch.cuda.get_device_name(0))

x = torch.randn(8192, 8192, device="cuda")
y = torch.randn(8192, 8192, device="cuda")
torch.cuda.synchronize()
t0 = time.time()
z = x @ y
torch.cuda.synchronize()
print(f"matmul 8192x8192: {time.time()-t0:.3f}s")

submit.sbatch:

#!/bin/bash
#SBATCH --job-name=pytorch-test
#SBATCH --partition=large
#SBATCH --gres=gpu:nvidia_rtx_pro_6000_blackwell_server_edition:1
#SBATCH --output=pytorch-test-%j.out
#SBATCH --error=pytorch-test-%j.err
#SBATCH --time=00:10:00

source /etc/profile.d/lmod.sh
export MODULEPATH=/mnt/apps/Modules/noarch/:/mnt/apps/Modules/linux-debian13-x86_64/:/mnt/apps/Modules/linux-debian13-zen4/
module load mamba-2.3.0-iiqe4tc
eval "$(mamba shell hook -s bash)"
mamba activate ~/mamba/<env-name>

python pytorch_test.py

sbatch submit.sbatch

Need a shell on the node while this job is running? See Attach a shell to a running job below.

Attach a shell to a running job

To get a shell on the node(s) running one of your jobs, find the job ID, then attach with --overlap:

squeue --me                              # find your JOBID
srun --jobid=<JOBID> --overlap --pty bash

--overlap is required: without it, Slurm tries to allocate a new step's worth of resources alongside the running job's step, which can hang or be rejected if the job isn't leaving anything spare. --overlap shares the existing allocation instead of competing with it. This works for jobs started either way — sbatch or srun.

Resources and Quota (QOS)

Default QOS

The default QOS is set at the partition level. Check which QOS is configured on a partition:

scontrol show partition <partition-name> | grep -i qos

Check the current limits set on that QOS:

sacctmgr show qos <qos-name> -P format=Name,MaxTRESPerUser

Group/Membership Access (`--account`)

--account=<team-name> binds a job to a team's resource allocation, taking precedence over any default.

Check assigned team(s):

sacctmgr show assoc user=$USER -P format=Account,User

Submit under the assigned team:

sbatch --account=<team-name> submit.sbatch

Specifying --account=<team-name> is sufficient on its own — Slurm resolves the appropriate limits automatically; no additional flag is required unless you're specifically instructed otherwise (see Advanced below).

Advanced: `--qos`

--qos=<name> explicitly overrides which QOS a job uses. Not required for normal account-based submission — only use it if specifically instructed to, e.g.:

sbatch --account=csg --qos=<qos-name> submit.sbatch