remote-server-ops

name	remote-server-ops
description	Use this skill when an agent needs to discover SSH targets, choose a remote server, sync changed local files, inspect remote software and GPU availability, start code on a remote server, monitor logs, and pull results back. Supports password or key authentication and both POSIX and Windows remote hosts.

Remote Server Ops

Use this skill when the user wants the agent to run code from the local workspace on a remote server.

This skill is built for agents. It should not depend on a Python environment. Prefer direct ssh, scp, rsync, tar, bash, and powershell flows. The existing tools/ directory may be read for defaults, but this skill must still work without it.

Trigger Conditions

Use this skill when the user asks to:

• connect to or inspect a remote server
• discover available SSH hosts from local SSH config files
• sync local code changes to a remote machine
• run training, evaluation, or scripts on a remote server
• choose an idle GPU before starting
• monitor remote logs or job status
• pull generated results back to local

Workflow

1. Resolve the server target from the local server store, the user's current message, or SSH config discovery.
2. Resolve authentication.
3. Detect remote OS and shell.
4. Run preflight checks, especially GPU availability.
5. Sync only the needed local changes.
6. Start the remote job.
7. Monitor status and logs.
8. Pull results back to local.

1. Resolve Server Targets

Prefer a local server store before probing SSH config files.

Server store:

• Use skills/remote-server-ops/servers.local.json as the local private server store.
• Keep servers.local.json out of git. It may contain passwords or machine-specific paths.
• Use servers.example.json as the schema example only.
• If creating or updating servers.local.json, write valid JSON and set owner-only permissions when the platform supports it:

chmod 600 skills/remote-server-ops/servers.local.json

Schema:

{
  "servers": [
    {
      "alias": "gpu-box",
      "host": "192.0.2.10",
      "port": 22,
      "user": "alice",
      "auth": "key",
      "key_path": "~/.ssh/id_ed25519",
      "remote_root": "/home/alice/project"
    }
  ]
}

Accepted fields:

• alias: short name the user can request later
• host: hostname or IP address
• port: SSH port, default 22
• user: SSH username
• auth: key or password
• key_path: private key path for auth=key
• password: password for auth=password
• local_root: optional local project root override
• remote_root: remote project root
• os: optional posix or windows hint

Resolution order:

1. If servers.local.json exists and contains matching servers, use it first.
2. If the user provides server details in natural language, extract alias, host, port, user, auth, password or key_path, and roots when present, then save or update that entry in servers.local.json by default.
3. If no stored or user-provided server is available, discover SSH targets from local SSH config files.

Default persistence policy:

• Save user-provided server details to the local private servers.local.json by default, including passwords for auth=password.
• Do not save only when the user explicitly says not to remember, not to save, or to use the credentials one time only.
• Do not print stored passwords in logs or summaries. Refer to them as present or missing.
• Do not commit servers.local.json or any real credentials.

JSON write rules:

• Never append raw text to servers.local.json.
• Read the existing JSON with a JSON parser, update the matching alias entry or append one new entry, then rewrite the whole file.
• Keep each server object in this field order when a field exists: alias, host, port, user, auth, key_path, password, local_root, remote_root, os.
• Use 2-space indentation.
• Write to a temporary file first, validate it as JSON, then move it over servers.local.json.
• After writing, run a JSON parse check such as:

python3 -m json.tool skills/remote-server-ops/servers.local.json >/dev/null

• If no JSON parser is available, do not update servers.local.json; ask the user to install python3, jq, or edit the file manually.
• Set owner-only permissions after each write when supported:

chmod 600 skills/remote-server-ops/servers.local.json

2. Discover SSH Targets

• First inspect local SSH config files, not only ~/.ssh/config.
• Use scripts/list-ssh-hosts.sh to list concrete Host entries.
• By default, search these locations when they exist:
  • $HOME/.ssh/config
  • /root/.ssh/config
  • /home/*/.ssh/config
  • /mnt/c/Users/*/.ssh/config
• The script output includes the config file path so the agent can tell the user where each alias came from.
• Ignore wildcard entries such as Host *.
• If multiple usable hosts exist and the user has not already chosen one, show the aliases and ask the user which one to use.
• If no usable host exists, or the user says not to use discovered SSH configs, ask the user for:
  • host
  • port
  • user
  • auth as password or key
  • password or key_path
  • local_root
  • remote_root

3. Authentication

• auth=key: use ssh -i <key_path> and scp -i <key_path>.
• auth=password: prefer sshpass if it is installed.
• If password auth is needed and sshpass is unavailable, tell the user the current shell cannot do a non-interactive password login safely and ask for either:
  • approval to use an interactive terminal flow
  • a private key
  • or a different machine that already trusts this host

When reading a Windows key from /mnt/c/..., verify it is an OpenSSH key rather than .ppk. If the key is PuTTY .ppk, tell the user it must be converted first.

4. Detect Remote OS

• Try a POSIX probe first: ssh ... 'uname -s'
• If that fails or clearly is not POSIX, try a PowerShell probe.
• Use references/posix-remote.md for Linux or other POSIX targets.
• Use references/windows-remote.md for Windows OpenSSH targets.

5. Preflight Checks

Before running any code, inspect:

• hostname
• current user
• target working directory
• disk free space
• Python availability if the project needs Python
• CUDA or GPU tooling if the task needs GPU
• nvidia-smi output if NVIDIA GPUs are present

GPU policy:

• You must choose an idle GPU before starting a new remote run.
• A GPU counts as unavailable if it already has compute processes attached, or if memory and utilization show obvious active use.
• If all GPUs are occupied, stop and tell the user the server is currently unavailable for a new run.

Use the GPU selection snippets in references/posix-remote.md or references/windows-remote.md.

6. Sync Policy

• Do not re-upload the whole project by default.
• If the local project is a git repo, use scripts/git-changed-files.sh to collect changed, untracked, and deleted files.
• Push only changed files unless the user explicitly asks for a full sync.
• Never push large datasets, checkpoints, or output directories unless the user asked for them.
• Do not delete remote files unless the user explicitly approves deletion.

Preferred order:

1. rsync with --files-from when available
2. scp for a small set of changed files
3. archive upload only when the first two are not practical

7. Start The Remote Job

For POSIX targets:

• Use nohup.
• Clear the previous nohup log before starting.
• Save logs to a stable path, for example .agent_runs/nohup.log.
• Save the PID to a stable path, for example .agent_runs/nohup.pid.
• Set CUDA_VISIBLE_DEVICES=<gpu_index> before launching.

For Windows targets:

• There is no native nohup.
• Use a PowerShell background process with stdout and stderr redirected to a stable log file.
• Preserve the same behavior: clear the old log first, record the new PID, and keep the log path stable.

The exact command forms are in the reference files.

8. Monitor

After starting:

• inspect the PID
• read the tail of the log
• check GPU usage on the selected GPU
• report failures immediately if the process exits early

Preferred checks:

• POSIX: ps -fp <pid>, tail -n 100 <log>, nvidia-smi
• Windows: Get-Process -Id <pid>, Get-Content -Tail 100 <log>, nvidia-smi

9. Pull Results Back

• Pull back the log file first.
• Then pull back only the run outputs the user cares about.
• Prefer run-specific output directories to avoid copying unrelated historical outputs.
• If the remote command created timestamps or run IDs, use them to scope the pullback.

Safety Rules

• Do not guess which server to use when multiple real choices exist. Ask.
• Do not commit servers.local.json or any real credentials.
• Do not echo stored passwords back to the user.
• Do not start on a busy GPU.
• Do not delete remote files or directories unless the user clearly requested that.
• Do not rely on tools/remote_ops.py being installed or runnable.
• If the local repo has no git metadata, ask before doing a broad sync.

Bundled Resources

• SSH host discovery: scripts/list-ssh-hosts.sh
• Local git change detection: scripts/git-changed-files.sh
• POSIX remote command patterns: references/posix-remote.md
• Windows remote command patterns: references/windows-remote.md