ncn-cluster

name

ncn-cluster

description

ncn cluster profile. Stand up a live link from this host into an HPC cluster (MIT Engaging / Slurm). Submit a Slurm job that runs Jupyter-on-GPU inside Apptainer on a compute node, then reach it through an ssh -L tunnel so the node's port appears at localhost. Used by the ncn conductor when the chosen node is a cluster; also use when the user asks to connect to the cluster, a compute node, or to get a remote GPU kernel.

ncn-cluster: Jupyter-on-GPU over Slurm

You are the host conductor. You do not live on the cluster; you reach in through a live SSH login session in the pane to your right, and a port-forward. Goal: make a GPU compute node's Jupyter port appear at localhost on this host.

 HOST (you, cxx)            LOGIN NODE              COMPUTE NODE (Slurm, GPU)
 conductor  ──ssh pane──▶   gateway   ──internal──▶ apptainer --nv → jupyter :PORT
 localhost:PORT ◀════════ ssh -L tunnel (two hops, one command: ncn tunnel) ═══════╝

Recover the connection details from this window's options:

SSH_PANE=$(tmux show-options -wqv @ncn_ssh_pane)   # the right pane
HOST=$(tmux show-options -wqv @ncn_host)           # login target (user@host)

Drive every cluster command through $SSH_PANE: tmux send-keys -t $SSH_PANE '<cmd>' Enter, then read back with tmux capture-pane -p -t $SSH_PANE. You are on the host, so you also hold the ssh creds: scp/ssh $HOST '<cmd>' directly when a step is non-interactive.

Your mode is in @ncn_mode. nc (connect) = just the login-node shell, for squeue/sacct/staging files; stop there, no job. ncn (endpoint) = the full flow below: submit a Jupyter-on-GPU job and tunnel its port to your localhost.

Parameters

Confirm with the user before submitting (offer env defaults, don't assume):

What	Env default	Example
partition	$ANU_CLUSTER_PART	mit_normal_gpu
image (.sif)	$ANU_CLUSTER_IMAGE	~/images/rapids.sif
gpus / cores / mem / walltime	(none)	--gres=gpu:1 -c8 --mem=32G -t04:00:00
port	(none)	8888

Moves

0. Confirm the session. capture-pane the right pane; make sure SSH is at a login-node shell prompt (not mid-auth/Duo, not a stale session).

1. Stage the job script (host → cluster, one move):

scp ~/.local/share/anu/plugins/ncn/skills/ncn-cluster/jlab.sbatch "$HOST":~/ncn-jlab.sbatch

2. Submit through the right pane, passing resources + image. This is the irreversible step (it consumes node-hours), so get a yes first:

NCN_IMAGE=<image> NCN_PORT=<port> sbatch -p <partition> --gres=gpu:1 -N1 -c8 --mem=32G -t04:00:00 ~/ncn-jlab.sbatch

Capture the returned job id.

3. Wait for RUNNING, then read where it landed (poll, don't spin tightly):

squeue -j <jobid> -h -o '%T %N'          # repeat until RUNNING
grep -m1 NCN_NODE ncn-jlab-<jobid>.log   # -> NCN_NODE=node1234
grep -m1 token=   ncn-jlab-<jobid>.log   # -> http://node1234:8888/lab?token=ab12cd...

Keep the token: it's the kernel's only access control on a shared node.

4. Open the tunnel from this host (collapses node:PORT → localhost:PORT through the login node in one command):

ncn tunnel <node> <port>     # e.g. ncn tunnel node1234 8888

5. Hand off. Report the local URL and let the user choose:

• Browser: http://localhost:<port>/lab?token=<token> on this host.
• Headless: a boxed cxc worker drives localhost:<port> over the Jupyter REST/WebSocket API with the token.

Teardown

• ncn down closes the tunnel pane (keeps the SSH session + job).
• Always scancel <jobid> when finished: an idle Jupyter job holds a GPU.

Failure modes

• No GPU in the kernel? Missing --nv, or the partition has no GPU.
• Tunnel drops on idle. ncn tunnel sets ServerAliveInterval; if the job ends the tunnel dies with it, so re-run after a fresh submit.
• Compute nodes are firewalled / no inbound ports. That's why the tunnel hops through the login node. Don't try to reach the node directly.
• Local validation has no GPU. A box on a Mac has no CUDA; use it for code/CPU paths, the cluster for real GPU work.