ssh-skill

name

ssh-skill

description

Operate SSH-accessible Linux hosts and clusters over the ssh-skill runtime: register servers, inspect and edit code, sync local↔remote, run commands, manage uv/conda environments, debug and publish services, drive remote AI coding CLIs (Claude Code, Gemini CLI, Cursor, OpenCode), and do cluster-profile-aware Slurm work (sbatch/squeue/sinfo), including login→debug jump-host clusters. Use whenever the task targets a remote machine or cluster over SSH. Triggers include SSH, ssh skill, 集群/cluster, Slurm, sbatch, srun, squeue, sinfo, 训练/提交任务/多卡, 添加/管理服务器, 同步代码/远程跑代码/看代码/配环境, 发布服务/看日志/远程排障, 登录节点/调试节点/跳板/ProxyJump, and 上科大/skd.

SSH Skill

Use this skill to operate the user's remote machine or cluster through the ssh-skill runtime, not through local-shell assumptions. The primary identity of this skill is generic SSH work: inspect code, edit config, sync files, register servers, run commands, debug services, and manage environments. When the remote target is a shared Slurm cluster, treat the cluster workflow as an add-on layer on top of SSH rather than the main identity of the skill.

Require runtime access

Assume the session has the ssh-skill runtime available, typically through:

• ssh_skill_server.py
• ssh_registry.py

The runtime should expose these baseline tools:

• ssh_list_hosts
• ssh_list_cluster_profiles
• ssh_get_cluster_profile
• ssh_list_ai_tool_profiles
• ssh_get_ai_tool_profile
• ssh_detect_ai_tools
• ssh_inspect_ai_workspace
• ssh_run_ai_tool
• ssh_sinfo
• ssh_squeue
• ssh_sacct
• ssh_slurm_job_info
• ssh_render_slurm_script
• ssh_sbatch_submit
• ssh_scancel
• ssh_exec
• ssh_upload
• ssh_download
• ssh_sync_dir
• ssh_list_conda_envs
• ssh_uv_sync

Prefer implementations that read standard local SSH aliases from ~/.ssh/config and the skill-managed server registry, instead of requiring a separate ad-hoc YAML inventory.

If richer file tools are available, prefer them for code and log inspection:

• ssh_list_dir
• ssh_find_files
• ssh_grep
• ssh_read_file
• ssh_write_file
• ssh_tail_file
• ssh_start_process
• ssh_check_process
• ssh_stop_process
• ssh_check_port

If these tools are unavailable, stop and tell the user that the ssh-skill runtime is not connected.

Works in Claude Code and Codex

This skill is agent-neutral. Two paths are referenced below:

• $SSH_SKILL_DIR — this skill's own directory, e.g. ~/.claude/skills/ssh-skill (Claude Code) or ~/.codex/skills/ssh-skill (Codex). Use it when invoking the bundled scripts.
• $SSH_SKILL_HOME — the per-user state directory. It defaults to <agent-config>/ssh-skill (~/.codex/ssh-skill for Codex, ~/.claude/ssh-skill for Claude Code) and can be overridden by setting the SSH_SKILL_HOME environment variable. It holds servers.json, managed_ssh_config, cluster_profiles/, and ai_tool_profiles/.

Manage servers first

This skill should remember the servers the user manages.

• The source of truth is the registry file $SSH_SKILL_HOME/servers.json.
• The effective SSH config generated by the skill is $SSH_SKILL_HOME/managed_ssh_config.
• The generated config should include the user's existing ~/.ssh/config so personal aliases and skill-managed aliases both work.
• Cluster-specific behavior should be attached through a reusable cluster_profile field on the server record.
• Site-specific cluster profiles should live in the user's local $SSH_SKILL_HOME/cluster_profiles/ directory instead of the public repository.
• Additional open-source profiles can be added as JSON files under $SSH_SKILL_HOME/cluster_profiles/.
• Remote AI coding tool behavior should stay modular too, through built-in or custom profiles under $SSH_SKILL_HOME/ai_tool_profiles/.

When the user wants to add, update, inspect, or remove managed servers, use:

• ssh_registry.py

Prefer these patterns:

python3 $SSH_SKILL_DIR/scripts/ssh_registry.py list
python3 $SSH_SKILL_DIR/scripts/ssh_registry.py add my-server --host 10.0.0.12 --user ubuntu --identity-file ~/.ssh/id_ed25519
python3 $SSH_SKILL_DIR/scripts/ssh_registry.py import-alias my-cluster --cluster-mode --scheduler slurm --cluster-profile my_lab_slurm
python3 $SSH_SKILL_DIR/scripts/ssh_registry.py update my-server --description "prod api" --tags prod api
python3 $SSH_SKILL_DIR/scripts/ssh_registry.py remove my-server

When the target is a Slurm cluster, record that in the registry with --cluster-mode --scheduler slurm, and prefer --cluster-profile <profile_id> when a reusable cluster profile exists.

For a jump-host cluster whose compute or debug nodes are only reachable through a login node, prefer the one-shot provision-cluster command instead of adding each host by hand. It reads the profile's access topology and registers every login and debug alias with the right ProxyJump and role:

# Registers sist-login1..3 (direct submit hosts) and sist-debug1..2 (ProxyJump through sist-login1).
python3 $SSH_SKILL_DIR/scripts/ssh_registry.py provision-cluster skd \
    --user <username> --identity-file ~/.ssh/<key>
# Preview without writing: add --dry-run

Use --role login / --role debug on add, update, or import-alias to tag a host's role manually. Debug or compute nodes (role=debug/role=compute) keep the cluster profile for context, but the runtime refuses to submit or cancel Slurm jobs from them; submit from a login node.

Follow this workflow

1. Identify the target host and goal.
   • If the server is not yet managed and the user wants it remembered, add it through ssh_registry.py before operating on it repeatedly.
   • Call ssh_list_hosts first when host aliases are unclear.
   • If the target is a known cluster, call ssh_get_cluster_profile or ssh_list_cluster_profiles before acting so host-specific Slurm rules are in context.
   • If the user wants a remote coding agent such as Claude Code, Gemini CLI, Cursor, or OpenCode, call ssh_detect_ai_tools or ssh_list_ai_tool_profiles before assuming the executable, instruction files, or auth method.
   • Confirm the remote repo path, working directory, environment name, output location, and expected result from existing context or remote inspection.
   • Read references/cluster-manual.md only when queue rules, storage paths, login policy, or Slurm conventions matter.
   • Read references/sist-ai-cluster.md when the target is the ShanghaiTech SIST / skd AI Cluster, or as a template for any login-to-debug jump-host Slurm cluster.
   • Read references/cluster-profiles.md when adding support for another cluster or extending the open-source profile system.
   • Read references/ai-coding-tools.md when remote AI CLI support, workspace instruction files, or custom AI tool profiles matter.
2. Build context safely.
   • Start with read-only inspection such as whoami, hostname, pwd, id, ls, git status, sinfo, squeue, module avail, conda env list, and nvidia-smi.
   • For remote AI coding, inspect the workspace with ssh_inspect_ai_workspace before assuming which helper tools or instruction files already exist.
   • Do not edit files, kill jobs, or overwrite remote state until the current situation is clear.
3. Pick the execution mode.
   • For the common workflow of local coding plus remote training, prefer this sequence: ssh_sync_dir -> ssh_list_conda_envs or ssh_uv_sync -> ssh_sinfo -> ssh_render_slurm_script -> ssh_sbatch_submit -> ssh_squeue -> ssh_tail_file -> ssh_sacct or ssh_slurm_job_info.
   • For the common workflow of remote AI-assisted code inspection, prefer this sequence: ssh_detect_ai_tools -> ssh_inspect_ai_workspace -> ssh_find_files or ssh_grep -> ssh_run_ai_tool(mode="analyze") -> direct SSH edits or sync steps after the situation is clear.
   • Use ssh_exec for inspection, Git operations, Slurm commands, package and environment commands, and other admin-light shell work.
   • Use ssh_get_cluster_profile to load built-in or custom cluster-specific rules, especially for hosts bound to reusable local profiles such as my_lab_slurm.
   • Use ssh_get_ai_tool_profile when the user names a specific remote coding CLI or when you want its instruction-file and auth expectations before invoking it.
   • Use ssh_run_ai_tool only after the workspace and safety mode are clear. Default to mode="analyze" so the remote agent stays read-only unless the user explicitly asked it to make changes.
   • Use ssh_sinfo before requesting resources such as 8 GPUs, so the partition and GPU layout are grounded in the current cluster state.
   • Use ssh_render_slurm_script to generate a batch script with explicit GPU, CPU, time, log, and environment settings.
   • Use ssh_sbatch_submit for actual job submission and ssh_squeue or ssh_sacct for structured follow-up.
   • Use ssh_slurm_job_info when a job is pending too long or lands in a strange state and you need the scheduler's detailed reason.
   • When the remote project uses Conda, prefer ssh_list_conda_envs first, then run ssh_exec or ssh_start_process with conda_env_name or conda_env_prefix.
   • Use ssh_sync_dir when the user has a local working tree that should be pushed to a remote path before running, debugging, or deploying.
   • Use ssh_uv_sync after code sync when the remote project uses uv; this creates or updates the remote environment without copying the local .venv.
   • Use ssh_find_files and ssh_grep for remote codebase discovery before falling back to hand-written shell search commands.
   • Use ssh_upload and ssh_download for source sync, configs, artifacts, checkpoints, and logs.
   • Use ssh_list_dir, ssh_read_file, ssh_write_file, and ssh_tail_file when available for code browsing, config changes, and log reading with less shell quoting overhead.
   • Use ssh_start_process, ssh_check_process, ssh_stop_process, and ssh_check_port for service-style debug runs.
   • Use sbatch for long-running workloads or anything that should survive disconnects.
   • Use srun only for short interactive debugging or one-off verification.
4. Execute and monitor.
   • Record host, remote path, command, job ID, queue, output log, and any published URL.
   • After sbatch, parse the job ID, watch squeue -j <jobid>, and inspect the first meaningful log output.
   • After launching a service, verify health from the cluster side before declaring success.
5. Report precisely.
   • Summarize what ran, where it ran, what changed, and what the next safe step is.
   • Surface blockers early: missing permission, missing module, wrong queue, quota, missing port, or policy conflict.

Respect these cluster rules

• Do not run heavy workloads on login nodes.
• Do not assume sudo, apt, or system-level package installation is allowed.
• Prefer module, Conda, source builds, or user-local installs.
• Do not assume Docker is supported. Prefer Singularity or Apptainer when containerization is required; build the image before uploading because there is usually no sudo inside containers either.
• On clusters with separate login and debug or compute nodes, reach the inner nodes by jumping through a login node. Provision them with provision-cluster so ProxyJump is wired automatically, then target the debug alias directly; the runtime handles the hop.
• Submit Slurm jobs only from login or submission nodes. Debug nodes are for code editing and environment debugging; the runtime blocks job submission from role=debug hosts.
• Do not hop from one login node to another; reconnect from your own client instead.
• Prefer the site mirror for package downloads when the cluster profile lists one.
• Do not SSH into compute nodes unless the user already has a running job there.
• Do not leave orphan processes on compute nodes.
• Do not use tmux or background jobs on compute nodes. Use tmux only on login nodes if needed.
• Always declare GPU usage explicitly for GPU jobs.
• Do not use rm, rm -rf, or ssh_sync_dir(delete=true) unless the user explicitly asked to delete remote files. Prefer Git-based rollback for code changes.
• Do not let a remote AI CLI make edits by default. Start with ssh_run_ai_tool(mode="analyze") and explicit human approval before switching to an execution-oriented prompt.
• Ask for confirmation before deleting remote files, canceling jobs, overwriting deployment directories, modifying shared paths, or restarting a user-facing service.

Handle common tasks

Inspect cluster state

• Start with ssh_list_hosts if the target alias is unknown.
• Use ssh_get_cluster_profile(host="<alias>") before assuming queue names or account rules on a managed Slurm host.
• Use sinfo and squeue -u "$USER" before making resource assumptions.
• When a job fails, inspect job state, exit code, log files, environment activation, current directory, and GPU visibility before changing code or config.

Sync code and artifacts

• Inspect the remote repo state before uploading a whole project.
• Prefer ssh_sync_dir for local source trees so rsync can push only the changed files.
• Keep local and remote environments separate: ssh_sync_dir should not copy .venv, venv, or other cache directories, and ssh_uv_sync should rebuild the remote environment in place.
• If the remote uses Conda, treat that environment as remote-only too; do not try to mirror a local Conda env bit-for-bit.
• If a rollback is needed, prefer Git commands over deleting files manually.
• Prefer incremental uploads for patches, config files, scripts, or checkpoints.
• Download logs and outputs when the user needs local inspection or archival.

Use remote AI coding CLIs

• Start with ssh_list_ai_tool_profiles or ssh_get_ai_tool_profile if the user explicitly asks for Claude Code, Gemini CLI, Cursor, or OpenCode support.
• Use ssh_detect_ai_tools to verify whether the remote host actually has the requested CLI, auth hints, and instruction files.
• Use ssh_inspect_ai_workspace to discover missing helper tools such as Git, ripgrep, Node.js, uv, conda, or Slurm commands before relying on a remote coding agent.
• Keep direct SSH file tools as the default path for deterministic work. Remote AI CLIs are optional add-ons, not a replacement for ssh_find_files, ssh_grep, ssh_read_file, ssh_sync_dir, or ssh_exec.
• Treat Cursor support here as the headless cursor-agent CLI. The local GUI editor remains outside the SSH runtime.
• When the remote AI CLI should stay read-only, use ssh_run_ai_tool(mode="analyze").
• When the user explicitly wants the remote AI CLI to act, use ssh_run_ai_tool(mode="execute"), then verify its changes with direct SSH inspection and Git state checks.

Run training or batch jobs

• Start with ssh_sinfo and ssh_get_cluster_profile before choosing a partition or asking for 8 GPUs.
• Use ssh_render_slurm_script to build a clean run.slurm, especially for multi-GPU training.
• Create or patch a run.slurm script in the project directory.
• Prefer explicit Slurm options for partition, time limit, CPU, memory, GPU, job name, output path, and working directory.
• Submit with ssh_sbatch_submit, then track with ssh_squeue, ssh_sacct, ssh_slurm_job_info, and the output log.
• Some institutional partitions require -A group-partition; inspect id <username> or your local docs if the group is unclear.
• If training fails, inspect the first meaningful log lines, the selected env, the exact launch command, GPU visibility, and the job's Slurm state before changing code.

Debug interactively

• Use srun for import tests, one-command reproductions, environment checks, port checks, and short shell sessions.
• Keep interactive sessions short and turn successful debugging steps into scripts or batch jobs when the work needs to continue.

Publish or restart a service

• Confirm the repo path, entry command, env activation, model or checkpoint path, desired port, visibility scope, and log location.
• Check whether the port is already in use and whether the chosen node or queue is appropriate.
• Prefer a reproducible launch command or batch script plus a stable log path.
• Verify process health, log output, and HTTP reachability before reporting success.
• Capture rollback information before replacing an existing service.

AI Coding Shortcuts

When doing remote coding work, prefer these tools in this order:

• Use ssh_find_files to discover likely files.
• Use ssh_grep to search symbols, config keys, stack traces, imports, and endpoint names.
• Use ssh_read_file to inspect specific files by line range.
• Use ssh_detect_ai_tools to check whether Claude Code, Gemini CLI, Cursor Agent CLI, or OpenCode is available on the current remote host.
• Use ssh_inspect_ai_workspace to spot missing shared instruction files and helper tools before handing the project to a remote AI CLI.
• Use ssh_run_ai_tool only after deciding whether the request should stay read-only or should actually modify the remote repo.
• Use ssh_get_cluster_profile to pull host-specific cluster rules when the managed server is a Slurm cluster.
• Use ssh_sinfo to inspect partitions or GPU availability before choosing 8-card or multi-node settings.
• Use ssh_render_slurm_script to generate or refresh run.slurm.
• Use ssh_sbatch_submit to submit the script and capture the job id.
• Use ssh_squeue, ssh_sacct, and ssh_slurm_job_info to understand queue state, runtime state, and failure metadata.
• Use ssh_list_conda_envs to discover available Conda envs on the remote host.
• Use ssh_exec with conda_env_name or conda_env_prefix when the command must run inside a Conda env.
• Use ssh_sync_dir to push a local repo or subdirectory to the remote machine before testing.
• Use ssh_uv_sync when the remote project relies on uv; let the remote host manage its own .venv.
• Use ssh_write_file for small config or script edits, and ssh_upload for larger patches or whole files.
• Use ssh_start_process to run a dev server or debug command in the background, optionally inside a Conda env.
• Use ssh_check_process, ssh_tail_file, and ssh_check_port to monitor that run.
• Use ssh_stop_process to cleanly stop the debug process.
• Use ssh_scancel only when the user explicitly asked to cancel a job.
• Treat rm as blocked by default inside ssh_exec and ssh_start_process; only enable destructive deletion when the user explicitly asked for it.

Prefer these command patterns

Use conservative, composable commands. Prefer commands like:

pwd
whoami
hostname
id
ls -lah
git status --short
sinfo
squeue -u "$USER"
module avail
module load <name>/<version>
conda env list
nvidia-smi
tail -n 200 <logfile>

Keep chained commands readable and recoverable. Avoid destructive one-liners unless the user has already approved them.

Load references only when needed

• Read references/cluster-manual.md when exact cluster rules, paths, queues, ports, or Slurm usage details matter.
• Read references/sist-ai-cluster.md for the ShanghaiTech SIST / 上科大 (skd) AI Cluster: its login-to-debug ProxyJump topology, one-command provision-cluster setup, no-sudo and Singularity rules, module list, and mirror URLs.
• Read references/cluster-profiles.md when profile ids, custom profile JSON files, or open-source extension rules matter.
• Read references/ssh-runtime.md when the task depends on the runtime contract, server registry layout, host aliases, or file transfer behavior.
• Read references/ai-coding-tools.md when you need the built-in AI tool profiles, common helper-tool recommendations, or the JSON schema direction for adding another remote coding CLI.