| name | ncn-cluster |
| description | ncn cluster profile. Stand up a live link from this host into an HPC cluster (MIT Engaging / Slurm). Submit a Slurm job that runs Jupyter-on-GPU inside Apptainer on a compute node, then reach it through an ssh -L tunnel so the node's port appears at localhost. Used by the ncn conductor when the chosen node is a cluster; also use when the user asks to connect to the cluster, a compute node, or to get a remote GPU kernel. |
ncn-cluster: Jupyter-on-GPU over Slurm
You are the host conductor. You do not live on the cluster; you reach in
through a live SSH login session in the pane to your right, and a port-forward.
Goal: make a GPU compute node's Jupyter port appear at localhost on this host.
HOST (you, cxx) LOGIN NODE COMPUTE NODE (Slurm, GPU)
conductor ──ssh pane──▶ gateway ──internal──▶ apptainer --nv → jupyter :PORT
localhost:PORT ◀════════ ssh -L tunnel (two hops, one command: ncn tunnel) ═══════╝
Recover the connection details from this window's options:
SSH_PANE=$(tmux show-options -wqv @ncn_ssh_pane)
HOST=$(tmux show-options -wqv @ncn_host)
Drive every cluster command through $SSH_PANE:
tmux send-keys -t $SSH_PANE '<cmd>' Enter, then read back with
tmux capture-pane -p -t $SSH_PANE. You are on the host, so you also hold the
ssh creds: scp/ssh $HOST '<cmd>' directly when a step is non-interactive.
Your mode is in @ncn_mode. nc (connect) = just the login-node shell, for
squeue/sacct/staging files; stop there, no job. ncn (endpoint) = the full
flow below: submit a Jupyter-on-GPU job and tunnel its port to your localhost.
Parameters
Confirm with the user before submitting (offer env defaults, don't assume):
| What | Env default | Example |
|---|
| partition | $ANU_CLUSTER_PART | mit_normal_gpu |
image (.sif) | $ANU_CLUSTER_IMAGE | ~/images/rapids.sif |
| gpus / cores / mem / walltime | (none) | --gres=gpu:1 -c8 --mem=32G -t04:00:00 |
| port | (none) | 8888 |
Moves
0. Confirm the session. capture-pane the right pane; make sure SSH is at a
login-node shell prompt (not mid-auth/Duo, not a stale session).
1. Stage the job script (host → cluster, one move):
scp ~/.local/share/anu/plugins/ncn/skills/ncn-cluster/jlab.sbatch "$HOST":~/ncn-jlab.sbatch
2. Submit through the right pane, passing resources + image. This is the
irreversible step (it consumes node-hours), so get a yes first:
NCN_IMAGE=<image> NCN_PORT=<port> sbatch -p <partition> --gres=gpu:1 -N1 -c8 --mem=32G -t04:00:00 ~/ncn-jlab.sbatch
Capture the returned job id.
3. Wait for RUNNING, then read where it landed (poll, don't spin tightly):
squeue -j <jobid> -h -o '%T %N'
grep -m1 NCN_NODE ncn-jlab-<jobid>.log
grep -m1 token= ncn-jlab-<jobid>.log
Keep the token: it's the kernel's only access control on a shared node.
4. Open the tunnel from this host (collapses node:PORT → localhost:PORT
through the login node in one command):
ncn tunnel <node> <port>
5. Hand off. Report the local URL and let the user choose:
- Browser:
http://localhost:<port>/lab?token=<token> on this host.
- Headless: a boxed
cxc worker drives localhost:<port> over the Jupyter
REST/WebSocket API with the token.
Teardown
ncn down closes the tunnel pane (keeps the SSH session + job).- Always
scancel <jobid> when finished: an idle Jupyter job holds a GPU.
Failure modes
- No GPU in the kernel? Missing
--nv, or the partition has no GPU.
- Tunnel drops on idle.
ncn tunnel sets ServerAliveInterval; if the job
ends the tunnel dies with it, so re-run after a fresh submit.
- Compute nodes are firewalled / no inbound ports. That's why the tunnel
hops through the login node. Don't try to reach the node directly.
- Local validation has no GPU. A
box on a Mac has no CUDA; use it for
code/CPU paths, the cluster for real GPU work.
