Remote SLURM cluster development via SSH. Use when running jobs, profiling, or developing on a remote SLURM cluster with pyxis/enroot containers. Covers SSH connection management, srun/sbatch/salloc job patterns, tmux-based allocation persistence, file transfer, and safe remote file access. Works with any SLURM cluster accessible via SSH.
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
Remote SLURM cluster development via SSH. Use when running jobs, profiling, or developing on a remote SLURM cluster with pyxis/enroot containers. Covers SSH connection management, srun/sbatch/salloc job patterns, tmux-based allocation persistence, file transfer, and safe remote file access. Works with any SLURM cluster accessible via SSH.
tags
["infrastructure","remote","slurm"]
license
Apache-2.0
metadata
{"author":"NVIDIA Corporation"}
Remote SLURM Development
Develop and execute on remote SLURM clusters via SSH commands. No filesystem
sync needed — all remote operations go through SSH.
Step 1: Resolve Cluster
Optional dependency — internal-env-info. Before doing
anything else in Step 1, check whether skills/internal-env-info/
exists in the toolkit. If it does not, skip every sub-step that invokes
the skill or reads its references/ files and proceed using only the
cluster fields the orchestrator wrote into job_spec.json (ssh_host,
slurm_user, remote_cwd, partition, account, ,
, , ). If a required field is also
missing, stop and ask the user to supply it — do emit an error
about the missing skill.
container_image
mounts
gpus_per_node
mfa_style
not
The orchestrator supplies one of two input shapes; pick the right path:
Explicit cluster — slurm_cluster name is provided. Skip
auto-selection and go straight to Load the per-cluster info.
Auto-select cluster — only hardware constraints are provided
(device_type, total_required_devices, required_devices_per_node).
Resolve a cluster from the constraints, then load its per-cluster info.
1a. Auto-select (only when no explicit cluster was provided)
Invoke the internal-env-info skill in constraint-based mode, passing
device_type, required_devices_per_node, and (optionally)
total_required_devices. That skill owns the filter rules and either
returns a single resolved cluster entry or stops and asks the user. Do not
implement the filter here.
Use node_count from job_spec.json directly as --nodes and
required_devices_per_node as --ntasks-per-node. Do not recompute
node_count — trtllm-case-executor is the single source of truth and
writes the resolved value into job_spec.json for every executor to read.
The returned entry's cluster_name, mfa_style, default_user_root_dir,
default_models_repo, and gpus_per_node feed Step 1b.
1b. Load the per-cluster info
Connection fields are read only from job_spec.json (written by
trtllm-case-executor): ssh_host, slurm_user, remote_cwd, account,
partition, mounts, container_image, gpus_per_node, and packages.
There is no per-cluster connection-config file to parse — if a required
field is absent from job_spec.json, stop and ask the user to supply it.
Do not silently default.
Invoke the internal-env-info skill in single-cluster mode with
the resolved cluster_name to retrieve per-cluster info: partition limits,
topology-aware scheduling guidance, default_models_repo,
default_user_root_dir, gpus_per_node, mfa_style (used by Step 2 to
choose the SSH path — see below), and shared constraint and tuning
patterns. The runtime container_image always comes from job_spec.json,
never from this skill.
Step 2: SSH Preflight
The cluster's mfa_style (returned by internal-env-info in Step 1b) is
the single source of truth for which SSH path to use. Switch on the
value — do not probe-and-fall-back on the SSH error string.
mfa_style
Path
Why
false
Direct (Step 2a)
Cluster accepts key-based / passwordless SSH; no special handling needed.
mfa_style not declared in the cluster reference (or internal-env-info is not installed). Probe direct first, fall back to MFA. When the reference exists, also ask the user to update it so future runs skip the probe; otherwise just continue silently.
2.0. Build <ssh_host> from slurm_user + cluster_name
Before any SSH attempt, derive the SSH destination. Priority order:
Caller-supplied ssh_host (passed in through case-executor) → use
verbatim, skip everything below. This is the canonical escape hatch.
internal-env-info skill installed → delegate for the
ssh_host_pattern key (defined in the skill's Default Environment Values
section). It returns a single-line snippet that concatenates slurm_user
and cluster_name into SSH_HOST. Do not re-derive or re-suffix
slurm_user here — internal-env-info already applied whatever its
slurm_user construction rule requires for that cluster.
Inputs to substitute:
slurm_user — pre-resolved from internal-env-info (already in its
final, SSH-ready form for that cluster). Read it from
job_spec.json.
cluster_name — from internal-env-info Step 1b.
Skill not installed → stop and ask the user to supply ssh_host
explicitly. Do not hard-code a login-host pattern in this skill —
the NVIDIA-internal convention lives only in internal-env-info.
The resolved value is the canonical <ssh_host> placeholder used everywhere
in the rest of this skill (Steps 2a/2b/2c, Step 3, recipes, file-transfer,
hang detection).
2a. Direct path — mfa_style: false
Verify connectivity using the user's existing SSH config:
ssh -o ConnectTimeout=5 <ssh_host> "echo ok"
If this succeeds, skip to Step 3. Do not create a trtllm-managed
ControlMaster — reuse whatever the user already has.
If it fails, do not silently switch to the MFA path. Stop and ask the
user to verify their ssh_host value or their key-based SSH setup.
A direct-path cluster failing here is a configuration issue, not an MFA
prompt to interpret.
2b. MFA path — mfa_style: true
First check whether a ControlMaster is already live (the user may have set
one up earlier in this session):
Otherwise the user must complete an interactive MFA handshake. MFA uses
browser-based device-code verification that needs an interactive SSH
session — the skill cannot complete this on the user's behalf because the
session must stay alive while the user authenticates in their browser and
presses ENTER.
If the cluster uses a device-code flow you will see a PIN and a URL like:
Authenticate with PIN XXXXXX at <your-identity-provider-device-url>
Open that URL, enter the PIN, approve, then press ENTER. The SSH session
will background itself as a persistent ControlMaster. Let me know when done.
After the user confirms, verify with
ssh -o ControlPath=/tmp/ssh-trtllm-%r@%h:%p -O check <ssh_host> and
proceed to Step 3.
2c. Unknown — mfa_style: null
The cluster reference (references/<cluster>.md) does not declare an
mfa_style value. Try the direct path (Step 2a) first. If it fails with
Permission denied (keyboard-interactive) — and only that — switch to
the MFA path (Step 2b). Any other failure stops here and is reported to
the user.
After connecting successfully, ask the user to update the cluster
reference file with the discovered value (mfa_style: true or
false, in the ## Authentication table) so that future runs skip
this probe and use the deterministic switch in Step 2a / 2b.
SSH Command Shorthand
Every SSH and SCP invocation in the rest of this skill uses the placeholders below. Both pin ControlPath to the trtllm-managed socket so they reuse the ControlMaster set up in Step 2 — without this, OpenSSH starts a fresh handshake on every call, which on MFA clusters re-prompts for keyboard-interactive auth (and defeats the whole point of multiplexing):
Placeholder
Expansion
<ssh_cmd>
ssh -o ControlPath=/tmp/ssh-trtllm-%r@%h:%p
<scp_cmd>
scp -o ControlPath=/tmp/ssh-trtllm-%r@%h:%p
When the user has their own ControlMaster configured in ~/.ssh/config (no trtllm-managed socket needed), the explicit -o ControlPath is harmless: if the path has no live master, OpenSSH falls back to a fresh connection. The connectivity probe in Step 2 Check 2 (ssh -o ConnectTimeout=5 <ssh_host> "echo ok") deliberately uses bare ssh to test un-multiplexed access; everywhere else use <ssh_cmd> / <scp_cmd>.
If remote_cwd does not exist, create it or ask the user. If tmux or SLURM
commands are not available, stop and inform the user.
Step 4: Prepare Remote Environment
This step sets up the user's working directory and clones the TensorRT-LLM
repository on the remote cluster. It runs once before any job submission.
MANDATORY: two-mechanism sync — both must run.
Tracked changes → temp branch (4c–4f): commit on a session-scoped
branch, push to origin, fetch + checkout on remote, delete the ref both
sides.
Untracked files (git ls-files --others --exclude-standard) →
tar | ssh pipe (4e.1), right after the remote checkout. Never touches
origin.
Do not shortcut with git fetch origin <repo_branch> && git checkout <repo_branch> —
repo_branch may not exist on origin, may be stale, or may lack uncommitted
edits. Always run 4c–4f and 4e.1.
Inputs received from trtllm-case-executor:
Field
Description
default_user_root_dir
Per-user root directory on the remote cluster (from exec-env-check / internal-env-info)
repo_url
Git remote URL of the local TensorRT-LLM repo (from git remote get-url origin). Used to clone the bare repository on the remote (Step 4b) and to push the temp branch to (Step 4d).
repo_branch
Informational only — the local current branch name (from git rev-parse --abbrev-ref HEAD). Used in Step 4c to remember which branch to restore locally after the temp branch is deleted. Never used as the ref the remote checks out — the remote always checks out the temp branch from Step 4c.
device_type
Required short GPU name (e.g., B200, B300). Used by Step 1a auto-selection. Ignored when an explicit cluster was provided.
total_required_devices
Total GPU processes across all nodes. Used by Step 1a auto-selection only. SLURM job sizing reads node_count from job_spec.json (precomputed by trtllm-case-executor) — do not recompute it here.
required_devices_per_node
Minimum GPUs per node. Used by Step 1a auto-selection (cluster must have gpus_per_node >= required_devices_per_node) and as --ntasks-per-node.
The actual ref the remote will check out is the temp branch created in 4c
and pushed in 4d; the remote checkout happens in 4e. Do not add a
git checkout <repo_branch> or git reset --hard origin/<repo_branch>
here as a shortcut — that path is forbidden (see the MANDATORY callout
at the top of Step 4).
4c. Create a temporary local branch capturing all local changes
Generate a unique temp branch name tied to this session. The branch name
must start with <user_name>_<model_name> so it is easy to identify
the owner and the workload it was created for. A timestamp suffix keeps
it unique across sessions:
# Resolve the user name (prefer the slurm_user input over whoami,# so the temp branch is attributed to the cluster user rather than the# host that happens to dispatch the job).
USER_NAME="${slurm_user:-$(whoami)}"# Resolve the workload name. job_spec.json's `model_name` is authoritative;# fall back to the repo basename if it is missing.
MODEL_NAME="${model_name:-$(basename "<REPO_ROOT>")}"# Sanitize to a git-ref-safe identifier (alphanum + `_`/`-` only).# Any other character is replaced with `_`.sanitize() { printf'%s'"$1" | tr -c 'A-Za-z0-9_-''_'; }
USER_NAME_SAFE=$(sanitize "$USER_NAME")
MODEL_NAME_SAFE=$(sanitize "$MODEL_NAME")
TEMP_BRANCH="${USER_NAME_SAFE}_${MODEL_NAME_SAFE}_$(date +%Y%m%d%H%M%S)"
Example resolved value: <user_name>_dwdp_deepseek_v3_lite_20260430145200.
Stash the current branch name so it can be restored later:
--allow-empty covers the no-tracked-changes case; the branch still exists
at HEAD and 4e.1 still runs.
4d. Push temp branch to the remote origin
Push the temp branch so the remote cluster can fetch it:
git -C "<REPO_ROOT>" push origin "$TEMP_BRANCH"
4e. Fetch and check out temp branch on the remote cluster
On the remote: stash prior-session leftovers, fetch, force-checkout the temp
branch, and re-materialize cpp/ from HEAD (stale cpp/ produces
missing-symbol ImportErrors in Step 5):
-z / --null keeps filenames with spaces or newlines intact; tar -x
auto-creates parent dirs; the guard skips the round-trip when nothing to
ship. Do not substitute rsync or sshfs (not reliably available on HPC).
4f. Delete the temp branch (local and remote)
After the remote checkout succeeds, immediately delete the temp branch from
the local repo and the remote origin, then restore the original local branch:
The remote cluster keeps the checked-out working tree from the deleted branch
— only the ref is removed, not the files.
4f.1 Create remote work_dir under the cloned repo
The remote work_dir must mirror the local layout (<REPO_ROOT>/work_dirs/<name>)
so that all per-run artefacts — slurm script, stdout/stderr logs, intermediate
files — live inside the cloned repo and are co-located with the code that
produced them.
remote_work_dir is precomputed by trtllm-case-executor (Step 1) and read
verbatim from job_spec.json — it is <remote_repo_path>/work_dirs/<basename(local_work_dir)>.
Do not fall back to <remote_cwd>/work_dirs/... or any other root; that
older layout is deprecated.
<ssh_cmd> <ssh_host> "mkdir -p <remote_work_dir>"
All subsequent steps (file uploads, sbatch submission, log paths) reference
<remote_work_dir> from job_spec.json.
4g. Copy build scripts to remote node
The skill ships two build scripts in scripts/ alongside this SKILL.md.
Copy them to a dedicated directory on the remote so they are accessible
from inside the SLURM container:
<LOCAL_SKILL_DIR> is the directory containing this SKILL.md file
(.claude/skills/exec-remote-slurm/).
Step 5: Build on Remote SLURM Node
Mandatory unless build_project: false. Step 4e wipes cpp/, so
the remote has no compiled C++ extensions until Step 5 succeeds (exit
code 0:0). Submitting any test/bench job before then fails on the
first import tensorrt_llm with missing-symbol errors.
Do not replace this with pip install -e . inside the user's
.slurm script — it does not reliably recompile cpp/, and when it
does it blocks the GPU allocation behind a 30+ minute compile. Step 5
runs the build as a 1-node side job and installs into the persistent
container volume.
If the caller sets build_project: false, skip Step 5 (the caller is
asserting the install is current). Do not silently downgrade true →
false because Step 5 looks expensive.
Submit a SLURM batch job that compiles TensorRT-LLM on the remote cluster
inside the official container. Wait for it to complete before proceeding to
job submission.
5a. Read the container image
The container image is resolved by trtllm-case-executor (Step 3) and
forwarded in job_spec.json as container_image. Read it from there —
do not grep current_image_tags.properties and do not re-derive from
the cluster config. case-executor is the single source of truth for the
image URI; this skill only consumes it.
Apply the enroot URI rewrite at use-time so the registry path is not
mistaken for a Docker-style credential lookup. The rewrite recipe lives
in the internal-env-info skill's Default Environment Values section
(enroot_uri_rewrite key):
Skill installed → delegate to it, passing the canonical
<container_image> from job_spec.json. The skill returns the bash
snippet (an sed over the registry-host prefix); run it here to
produce <IMAGE_PATH>.
Skill not installed → use <container_image> verbatim as <IMAGE_PATH>.
Use <IMAGE_PATH> as the value passed to --container-image= in the
build sbatch below and in every srun/sbatch recipe later in this
skill. The canonical (un-rewritten) URI in job_spec.json is unchanged;
the rewrite is purely a transport adaptation for enroot.
5b. Derive build architecture
Map device_type (from the env-check output) to the build_wheel.py-a flag:
Runs inside container — calls build_wheel.py with defaults
Both scripts accept the same positional arguments:
<container_image> <user_root_dir> <scripts_dir> <repo_dir> [build_wheel_args...]
Recipes
Pick the recipe that matches your workflow. Each recipe is self-contained —
copy it, substitute the <placeholders> from config, and run.
Placeholders (all from job_spec.json unless noted):
Placeholder
Source
<ssh_host>, <account>, <partition>, <mounts>
job_spec.json
<image>
container_image from job_spec.json (resolved by trtllm-case-executor Step 3). Apply enroot rewrite (/ → #) at use-time per Step 5a — substitute the rewritten <IMAGE_PATH> here. Do not re-resolve from the cluster config or current_image_tags.properties.
<job_name>
Pre-resolved by trtllm-case-executor and forwarded as job_name in job_spec.json. Substitute directly into -J <job_name>; do not reconstruct from account / subproject / detail.
<nodes>
node_count from job_spec.json (computed by trtllm-case-executor). Do not recompute.
<gpus>
required_devices_per_node from job_spec.json — probe only if absent (see bottom of this section)
<remote_repo_path>
remote_repo_path from job_spec.json — <default_user_root_dir>/<repo_dir_name> precomputed by trtllm-case-executor Step 1. The cloned repo and --container-workdir for srun. Do not recompute.
<remote_work_dir>
remote_work_dir from job_spec.json — <remote_repo_path>/work_dirs/<basename(local_work_dir)> precomputed by trtllm-case-executor Step 1. Used for --output/--error slurm log paths and any per-run artefact uploads. Do not recompute or fall back to <remote_cwd>/work_dirs/....
<time>
Walltime in HH:MM:SS
Job naming: Every recipe includes -J <job_name> using the
pre-resolved job_name value from job_spec.json (built by
trtllm-case-executor). This is required — without -J, SLURM emits
warnings and jobs are hard to identify in squeue. Do not reconstruct
the name from account / subproject / detail here; that convention
lives in trtllm-case-executor.
GPU allocation: Cluster behavior varies:
Exclusive-access clusters (most NVIDIA internal): Nodes are allocated
exclusively with all GPUs available. No --gpus-per-node needed.
Shared/QOS clusters: Require --gpus-per-node=<gpus_per_node>. Without
it, you may get a CPU-only node (no GPUs at all), a "Cannot find GPU
specification" error, or a "QOSMinGRES" error (below minimum).
If gpus_per_node is set in the cluster config, add --gpus-per-node=<gpus_per_node>
to every srun, salloc, and sbatch command. If not set, omit it.
If a job fails with GPU-related errors and gpus_per_node is not configured,
probe the node's GPU count, set gpus_per_node in the cluster config, and retry.
Do not set CUDA_VISIBLE_DEVICES for MPI workloads — the MPI runtime handles
GPU assignment.
Recipe 1: Single-process command (srun)
One-off commands on a single node: profiling, inspecting, lightweight scripts.
Framework-launched distributed training (PyTorch DDP, DeepSpeed). SLURM starts
<nodes> × <gpus> processes. Pyxis/enroot sets RANK, LOCAL_RANK, and
WORLD_SIZE. Each process runs the training script directly.
Interactive development with persistent container state. Use this when you need
multiple commands on the same allocation — installing packages, compiling,
running experiments iteratively.
Relationship to persistent_mode: The persistent_mode field in
job_spec.json is consumed only by exec-local-slurm. This
skill does not read persistent_mode, container_name, or
container_mounts from job_spec.json for allocation reuse — remote
persistence is driven by this recipe (tmux + named salloc + named
container) and controlled by inputs to this skill directly (e.g.,
release_allocation, alloc_time_limit). If a caller forwards a
persistent_mode=true job spec into a remote_slurm run, treat the field
as informational only and use this recipe's flow.
Step 1: Allocate via tmux
tmux keeps the allocation alive on the frontend — invisible plumbing.
Add --container-writable if the workload writes to container paths outside
of mounted directories (e.g., pip install into site-packages).
Step 5: Release the allocation
Clean up the named container, then kill tmux. If the allocation has already
expired (walltime exceeded), the srun will fail — that is fine, the container
is already gone. Just kill the tmux session.
Never use cat on remote files unless you are certain the file is small.
Never run heavy computation on the frontend
The frontend node is shared. Only use it for SLURM commands, lightweight file
operations, and tmux management.
Check before creating
Before creating resources, check if they already exist:
SSH ControlMaster: ssh -O check <ssh_host> 2>&1 || <ssh_cmd> -O check <ssh_host> (probe both the user's default socket and the trtllm-managed socket from Step 2)
%S = estimated start time. Parse it to calculate wait.
If RUNNING → proceed to next step.
If PENDING and wait ≤ 10 minutes → wait for it, proceed.
If PENDING and wait > 10 minutes (or no start time shown):
a. Infer how long you actually need from context (profiling → ~30 min,
iterative dev → ~2h, training run → match expected duration).
Round up to the next hour boundary.
b. Reduce walltime in-place:
c. Tell the user: "Reduced walltime from Xh to Yh for faster scheduling —
you'll have less time on the allocation."
d. Wait ~15 seconds, re-check estimated start time.
e. If still > 10 minutes: inform user with the ETA and suggest
alternatives — different partition (e.g., backfill), fewer nodes.
Hang Detection Monitoring
When monitoring a running job (polling via squeue), also check the remote log
file every 60 seconds for the pattern hang detected (case-insensitive). This
catches GPU hangs, NCCL deadlocks, and other stalls that keep the job running
indefinitely without making progress.
Check for hang (run alongside each squeue poll once job is RUNNING):
Check the reason (%r) and estimated start (%S). If wait > 10 minutes,
follow the walltime negotiation flow (see Walltime strategy). If already
negotiated and still stuck, suggest alternatives: backfill partition, fewer
nodes, or a different cluster.
Job fails: Read last lines, never dump full output:
Hang detected: If the log file contains hang detected, the job is
auto-cancelled (see Hang Detection Monitoring section above). Report
HANG_DETECTED with the last 200 lines of log context.
