exec-remote-slurm

# Remote SLURM Development Develop and execute on remote SLURM clusters via SSH commands. No filesystem sync needed — all remote operations go through SSH. ## Step 1: Resolve Cluster **Optional dependency — `internal-env-info`.** Before doing anything else in Step 1, check whether `skills/internal-env-info/` exists in the toolkit. If it does **not**, skip every sub-step that invokes the skill or reads its `references/` files and proceed using only the cluster fields the orchestrator wrote into `job_spec.json` (`ssh_host`, `slurm_user`, `remote_cwd`, `partition`, `account`, `container_image`, `mounts`, `gpus_per_node`, `mfa_style`). If a required field is also missing, stop and ask the user to supply it — do **not** emit an error about the missing skill. The orchestrator supplies one of two input shapes; pick the right path: - **Explicit cluster** — `slurm_cluster` name is provided. Skip auto-selection and go straight to **Load the per-cluster info**. - **Auto-select cluster** — only hardware constraints are provided (`device_type`, `total_required_devices`, `required_devices_per_node`). Resolve a cluster from the constraints, then load its per-cluster info. ### 1a. Auto-select (only when no explicit cluster was provided) Invoke the `internal-env-info` skill in **constraint-based mode**, passing `device_type`, `required_devices_per_node`, and (optionally) `total_required_devices`. That skill owns the filter rules and either returns a single resolved cluster entry or stops and asks the user. Do not implement the filter here. Use `node_count` from `job_spec.json` directly as `--nodes` and `required_devices_per_node` as `--ntasks-per-node`. **Do not recompute `node_count`** — `trtllm-case-executor` is the single source of truth and writes the resolved value into `job_spec.json` for every executor to read. The returned entry's `cluster_name`, `mfa_style`, `default_user_root_dir`, `default_models_repo`, and `gpus_per_node` feed Step 1b. ### 1b. Load the per-cluster info Connection fields are read **only** from `job_spec.json` (written by `trtllm-case-executor`): `ssh_host`, `slurm_user`, `remote_cwd`, `account`, `partition`, `mounts`, `container_image`, `gpus_per_node`, and `packages`. There is no per-cluster connection-config file to parse — if a required field is absent from `job_spec.json`, stop and ask the user to supply it. Do not silently default. Invoke the `internal-env-info` skill in single-cluster mode with the resolved `cluster_name` to retrieve per-cluster info: partition limits, topology-aware scheduling guidance, `default_models_repo`, `default_user_root_dir`, `gpus_per_node`, `mfa_style` (used by Step 2 to choose the SSH path — see below), and shared constraint and tuning patterns. The runtime `container_image` always comes from `job_spec.json`, never from this skill. ## Step 2: SSH Preflight The cluster's `mfa_style` (returned by `internal-env-info` in Step 1b) is the **single source of truth for which SSH path to use**. Switch on the value — do **not** probe-and-fall-back on the SSH error string. | `mfa_style` | Path | Why | |-------------|------|-----| | `false` | Direct (Step 2a) | Cluster accepts key-based / passwordless SSH; no special handling needed. | | `true` | MFA (Step 2b) | Cluster requires interactive MFA (Duo / device-code / one-time PIN). | | `null` | Unknown (Step 2c)| `mfa_style` not declared in the cluster reference (or `internal-env-info` is not installed). Probe direct first, fall back to MFA. When the reference exists, also ask the user to update it so future runs skip the probe; otherwise just continue silently. | ### 2.0. Build `<ssh_host>` from `slurm_user` + `cluster_name` Before any SSH attempt, derive the SSH destination. Priority order: 1. **Caller-supplied `ssh_host`** (passed in through case-executor) → use verbatim, skip everything below. This is the canonical escape hatch. 2. **`internal-env-info` skill installed** → delegate for the `ssh_host_pattern` key (defined in the skill's `Default Environment Values` section). It returns a single-line snippet that concatenates `slurm_user` and `cluster_name` into `SSH_HOST`. Do **not** re-derive or re-suffix `slurm_user` here — `internal-env-info` already applied whatever its `slurm_user` construction rule requires for that cluster. Inputs to substitute: - `slurm_user` — pre-resolved from `internal-env-info` (already in its final, SSH-ready form for that cluster). Read it from `job_spec.json`. - `cluster_name` — from `internal-env-info` Step 1b. 3. **Skill not installed** → stop and ask the user to supply `ssh_host` explicitly. Do **not** hard-code a login-host pattern in this skill — the NVIDIA-internal convention lives only in `internal-env-info`. The resolved value is the canonical `<ssh_host>` placeholder used everywhere in the rest of this skill (Steps 2a/2b/2c, Step 3, recipes, file-transfer, hang detection). ### 2a. Direct path — `mfa_style: false` Verify connectivity using the user's existing SSH config: ```bash ssh -o ConnectTimeout=5 <ssh_host> "echo ok" ``` If this succeeds, **skip to Step 3**. Do not create a trtllm-managed ControlMaster — reuse whatever the user already has. If it fails, do **not** silently switch to the MFA path. Stop and ask the user to verify their `ssh_host` value or their key-based SSH setup. A direct-path cluster failing here is a configuration issue, not an MFA prompt to interpret. ### 2b. MFA path — `mfa_style: true` First check whether a ControlMaster is already live (the user may have set one up earlier in this session): ```bash ssh -O check <ssh_host> 2>&1 || ssh -o ControlPath=/tmp/ssh-trtllm-%r@%h:%p -O check <ssh_host> 2>&1 ``` If either succeeds, **skip to Step 3** — reuse it. Otherwise the user must complete an interactive MFA handshake. MFA uses browser-based device-code verification that needs an interactive SSH session — the skill cannot complete this on the user's behalf because the session must stay alive while the user authenticates in their browser and presses ENTER. Tell the user to run the ControlMaster command: > Run this in your terminal: > > ssh -fNM -o ControlMaster=auto -o ControlPath=/tmp/ssh-trtllm-%r@%h:%p -o ControlPersist=24h <ssh_host> > > If the cluster uses a device-code flow you will see a PIN and a URL like: > > Authenticate with PIN XXXXXX at <your-identity-provider-device-url> > > Open that URL, enter the PIN, approve, then press ENTER. The SSH session > will background itself as a persistent ControlMaster. Let me know when done. After the user confirms, verify with `ssh -o ControlPath=/tmp/ssh-trtllm-%r@%h:%p -O check <ssh_host>` and proceed to Step 3. ### 2c. Unknown — `mfa_style: null` The cluster reference (`references/<cluster>.md`) does not declare an `mfa_style` value. Try the direct path (Step 2a) first. If it fails with `Permission denied (keyboard-interactive)` — and only that — switch to the MFA path (Step 2b). Any other failure stops here and is reported to the user. After connecting successfully, **ask the user to update the cluster reference file** with the discovered value (`mfa_style: true` or `false`, in the `## Authentication` table) so that future runs skip this probe and use the deterministic switch in Step 2a / 2b. ## SSH Command Shorthand Every SSH and SCP invocation in the rest of this skill uses the placeholders below. Both pin `ControlPath` to the trtllm-managed socket so they reuse the ControlMaster set up in Step 2 — without this, OpenSSH starts a fresh handshake on every call, which on MFA clusters re-prompts for keyboard-interactive auth (and defeats the whole point of multiplexing): | Placeholder | Expansion | |-------------|-----------| | `<ssh_cmd>` | `ssh -o ControlPath=/tmp/ssh-trtllm-%r@%h:%p` | | `<scp_cmd>` | `scp -o ControlPath=/tmp/ssh-trtllm-%r@%h:%p` | When the user has their own ControlMaster configured in `~/.ssh/config` (no trtllm-managed socket needed), the explicit `-o ControlPath` is harmless: if the path has no live master, OpenSSH falls back to a fresh connection. The connectivity probe in Step 2 Check 2 (`ssh -o ConnectTimeout=5 <ssh_host> "echo ok"`) deliberately uses bare `ssh` to test un-multiplexed access; everywhere else use `<ssh_cmd>` / `<scp_cmd>`. ## Step 3: Remote Environment Check Verify the remote environment is ready: ```bash <ssh_cmd> <ssh_host> "test -d <remote_cwd> && echo ok" <ssh_cmd> <ssh_host> "which tmux" <ssh_cmd> <ssh_host> "squeue --version" ``` If `remote_cwd` does not exist, create it or ask the user. If tmux or SLURM commands are not available, stop and inform the user. ## Step 4: Prepare Remote Environment This step sets up the user's working directory and clones the TensorRT-LLM repository on the remote cluster. It runs once before any job submission. > **MANDATORY: two-mechanism sync — both must run.** > - **Tracked changes** → temp branch (**4c–4f**): commit on a session-scoped > branch, push to `origin`, fetch + checkout on remote, delete the ref both > sides. > - **Untracked files** (`git ls-files --others --exclude-standard`) → > `tar | ssh` pipe (**4e.1**), right after the remote checkout. Never touches > `origin`. > > **Do not** shortcut with `git fetch origin <repo_branch> && git checkout <repo_branch>` — > `repo_branch` may not exist on `origin`, may be stale, or may lack uncommitted > edits. Always run 4c–4f **and** 4e.1. **Inputs received from `trtllm-case-executor`:** | Field | Description | |-------|-------------| | `default_user_root_dir` | Per-user root directory on the remote cluster (from `exec-env-check` / `internal-env-info`) | | `repo_url` | Git remote URL of the local TensorRT-LLM repo (from `git remote get-url origin`). **Used to clone the bare repository on the remote (Step 4b) and to push the temp branch to (Step 4d).** | | `repo_branch` | **Informational only** — the local current branch name (from `git rev-parse --abbrev-ref HEAD`). Used in Step 4c to remember which branch to restore locally after the temp branch is deleted. **Never used as the ref the remote checks out** — the remote always checks out the temp branch from Step 4c. | | `device_type` | Required short GPU name (e.g., `B200`, `B300`). Used by Step 1a auto-selection. Ignored when an explicit cluster was provided. | | `total_required_devices` | Total GPU processes across all nodes. Used by Step 1a auto-selection only. SLURM job sizing reads `node_count` from `job_spec.json` (precomputed by `trtllm-case-executor`) — do not recompute it here. | | `required_devices_per_node` | Minimum GPUs per node. Used by Step 1a auto-selection (cluster must have `gpus_per_node >= required_devices_per_node`) and as `--ntasks-per-node`. | **Procedure:** ### 4a. Create user root directory ```bash <ssh_cmd> <ssh_host> "mkdir -p <default_user_root_dir>" ``` If `default_user_root_dir` is null or empty, skip this step and use `remote_cwd` directly. ### 4b. Clone or update the repository Determine the repo directory name from `repo_url` (strip `.git` suffix if present): ```bash REPO_DIR_NAME=$(basename <repo_url> .git) REMOTE_REPO_PATH="<default_user_root_dir>/${REPO_DIR_NAME}" ``` Check if it already exists: ```bash <ssh_cmd> <ssh_host> "test -d <remote_repo_path>/.git && echo exists" ``` If the repository does **not** exist, clone it: ```bash <ssh_cmd> <ssh_host> "cd <default_user_root_dir> && git clone <repo_url>" ``` If the repository **already exists**, refresh remote refs only — do **not** checkout or reset to any branch here: ```bash <ssh_cmd> <ssh_host> "cd <remote_repo_path> && git fetch origin --prune" ``` The actual ref the remote will check out is the temp branch created in 4c and pushed in 4d; the remote checkout happens in 4e. **Do not** add a `git checkout <repo_branch>` or `git reset --hard origin/<repo_branch>` here as a shortcut — that path is forbidden (see the MANDATORY callout at the top of Step 4). ### 4c. Create a temporary local branch capturing all local changes Generate a unique temp branch name tied to this session. The branch name **must start with `<user_name>_<model_name>`** so it is easy to identify the owner and the workload it was created for. A timestamp suffix keeps it unique across sessions: ```bash # Resolve the user name (prefer the slurm_user input over whoami, # so the temp branch is attributed to the cluster user rather than the # host that happens to dispatch the job). USER_NAME="${slurm_user:-$(whoami)}" # Resolve the workload name. job_spec.json's `model_name` is authoritative; # fall back to the repo basename if it is missing. MODEL_NAME="${model_name:-$(basename "<REPO_ROOT>")}" # Sanitize to a git-ref-safe identifier (alphanum + `_`/`-` only). # Any other character is replaced with `_`. sanitize() { printf '%s' "$1" | tr -c 'A-Za-z0-9_-' '_'; } USER_NAME_SAFE=$(sanitize "$USER_NAME") MODEL_NAME_SAFE=$(sanitize "$MODEL_NAME") TEMP_BRANCH="${USER_NAME_SAFE}_${MODEL_NAME_SAFE}_$(date +%Y%m%d%H%M%S)" ``` Example resolved value: `<user_name>_dwdp_deepseek_v3_lite_20260430145200`. Stash the current branch name so it can be restored later: ```bash ORIGINAL_BRANCH=$(git -C "<REPO_ROOT>" rev-parse --abbrev-ref HEAD) ``` Create the temp branch at HEAD and stage tracked changes only (`-u`, not `-A` — untracked files ship in 4e.1): ```bash git -C "<REPO_ROOT>" checkout -b "$TEMP_BRANCH" git -C "<REPO_ROOT>" add -u git -C "<REPO_ROOT>" commit --allow-empty -m "sync: temp snapshot for remote execution" ``` `--allow-empty` covers the no-tracked-changes case; the branch still exists