Interactive Job
Run interactive workloads on the HPC with VS Code Remote Tunnels, Jupyter Notebooks and X11 VNC sessions.
Unreviewed migration
The native interactive mode mechanism of the job scheduler does not work. Do NOT try commands like qrsh or qlogin.
VS Code
VS Code stores data in the user home directory, which is shared across the CSE infrastructure (VLAB, HPC, etc.). Therefore, ensure that you do not run multiple instances of VS Code in different locations simultaneously. If you are or have been running VS Code on VLAB or the login nodes, make sure to exit cleanly and stop all existing processes before proceeding.
While VS Code supports a normal Microsoft account, UNSW SSO is NOT supported. To avoid confusion, here we cover GitHub only.
To establish a remote tunnel session on a compute node, submit the following job:
#!/bin/bash -l
# PLEASE MAKE SURE YOU CHANGE THE FOLLOWING QSUB SETTINGS TO SUIT YOUR NEEDS
#
#$ -j y
#$ -e $JOB_ID_$JOB_NAME.out
#$ -o $JOB_ID_$JOB_NAME.out
#$ -cwd
#$ -N vscode-tunnel
#$ -l mem=15G,jobfs=10G
#$ -V
#
module load vscode/vscode-cli
launch_tunnelThe vscode CLI prints out the instruction in the job's output file, like below. Simply follow the link to get authenticated.
A GitHub account is required for authentication to access the established Remote Tunnels.
$ cat 534_vscode-tunnel.out
Loading vscode - vscode-cli
spawn -ignore HUP /share/apps/noarch/vscode/vscode-cli/code tunnel
*
* Visual Studio Code Server
*
* By using the software, you agree to
* the Visual Studio Code Server License Terms (https://aka.ms/vscode-server-license) and
* the Microsoft Privacy Statement (https://privacy.microsoft.com/en-US/privacystatement).
*
✔ How would you like to log in to Visual Studio Code? · GitHub Account
To grant access to the server, please log into https://github.com/login/device and use code EFFF-CE86
Open this link in your browser https://vscode.dev/tunnel/wolfpack_zxxxxxx/import/xxxx/x/zxxxxxx/Once the job prints out the URL, it is ready for connection. To connect, open VS Code on your local computer, open the Command Palette (F1) and run the command Remote Tunnels. Follow the prompts:
For more detailed information, please check out the official documentation: Remote Tunnels extension.
GitHub issue
It has been observed that VS Code sometimes fails to establish a session after successful authentication against GitHub (the 'Open this link' message never appears). The cause of this issue is unclear. Possible solutions include:
- Logging out of GitHub and logging in again
- Unregistering all tunnels that VS Code has saved via Remote Explorer (that's right, ALL of them)
Reset VS Code
In case things are broken, please try a full reset:
qdelthe job which is running VS Code- Delete any old processes on any other CSE systems (VLAB, lab computer etc)
- Unregister the broken tunnel
- Delete the corrupted user data:
rm -rf ~/.vscode*
Jupyter Notebooks
The easier way to run Jupyter Notebooks is via VS Code:
code.visualstudio.com/docs/datascience/jupyter-notebooks
X11 VNC
This process essentially uses SSH tunnelling. If you're unfamiliar with SSH tunnelling, this resource may be helpful: blog.mobatek.net — SSH tunnels and port forwarding.
A graphical interactive session can be obtained by running an x11vnc container. The session can be accessed via the web browser (or VNC client) on the user's local computer, with support for NVIDIA CUDA.
Please use the following submission script as a starting point:
#!/bin/bash -l
#
# PLEASE MAKE SURE YOU CHANGE THE FOLLOWING QSUB SETTINGS TO SUIT YOUR NEEDS
#
#$ -j y
#$ -e $JOB_ID_$JOB_NAME.out
#$ -o $JOB_ID_$JOB_NAME.out
#$ -cwd
#$ -N x11_vnc
#$ -l mem=15G,jobfs=180G # adding more for GPGPU etc etc
#$ -V
#$ -P project_name
module load vnc/x11vnc-nvidia
# By default (if IMAGE_FILE_NAME is not provided), the newest cuda image is used. Other images are available at
# /share/apps/sif/x11vnc/
# Just provide the image file name as the first argument ($1)
start-vncserver [IMAGE_FILE_NAME]Submit this script via qsub. The VNC connection information will be available in the job output log and will look like this:
$ cat 295_x11_vnc.out
Loading vnc - x11vnc-nvidia
******** SSH TUNNEL INFO ********
You must setup SSH tunneling on your local computer for connecting.
For Windows:
https://blog.mobatek.net/post/ssh-tunnels-and-port-forwarding/
For Linux or macOS, sample ssh below:
ssh -L PORT:wp-delta-f03.cse.unsw.edu.au:PORT login.cse.unsw.edu.au
PORT for web browser: provided by the HTTP URL below
>>OR<<
PORT for VNC client: 59xx (see VNC viewer below)
Running: apptainer run -B /import/reed/8/z3155686 -B /scratch_local/295.1.all.q /share/apps/sif/x11vnc/12.6.1-cudnn-devel-ubuntu22.04-x11vnc.sif
*** Running /sbin/setuser ubuntu $DOCKER_CMD...
Open your web browser with URL:
http://localhost:6080/vnc.html?resize=downscale&autoconnect=1&password=wxaZpA1o
or connect your VNC viewer to localhost:5900 with password wxaZpA1oOnce SSH tunnelling is set up correctly, the interactive session can be accessed via the browser, like:
Bind paths and mounts
By default, the startup script binds $HOME and $TMPDIR to the container. Typically these two paths are available inside the container out of the box:
/import/reed/8/zID(this is an example)/scratch_local/1563.1.all.q(this is an example)
Additional directories can be added to the container by using an env variable:
# This is required BEFORE submitting the job, or simply put it in .bashrc if it does not change often
export APPTAINER_BINDPATH="/PATH1,/PATH2"CUDA samples
We can run CUDA samples for verification purposes. Run the following inside the interactive session:
The compilation will not fully complete, but there are enough CUDA binaries that can be used for testing.
# switch to the temporary job scratch space on the compute node's local NVMe/SSD storage.
cd /scratch_local/295.1.all.q
git clone https://github.com/NVIDIA/cuda-samples.git
cd cuda-samples
make