infrastructure-ops

> **IMPORTANT: EXECUTION CONTEXT** > This skill describes *what to do* — domain patterns and decision-making. > It does NOT describe *how* to call the API. > > 1. **If you have `wsh_*` tools** (check your toolkit for `wsh_send_input`, > `wsh_get_screen`, etc.): use them directly. Operation names in this > skill generally map to tool names (e.g., "send input" → `wsh_send_input`). > When in doubt, list your available `wsh_*` tools. > 2. **If you do NOT have `wsh_*` tools**: you are in HTTP/curl fallback mode. > **DO NOT GUESS endpoints or CLI subcommands.** > Load the full API reference first: search your workspace for > `skills/core/` and read `SKILL.md`. It contains every endpoint > with working curl examples and a bootstrap sequence. > 3. **Quick bootstrap**: `curl -sf --unix-socket ${XDG_RUNTIME_DIR:-/tmp}/wsh/default.http.sock http://localhost/health` > — if that fails: `wsh server -L agent-$$ --ephemeral &` and retry. # wsh:infrastructure-ops -- Fleet Management Ansible, Puppet, and Chef exist because you can't interactively operate 50 machines at once. They solve this with declarative configs and idempotent operations -- you describe the desired state, the tool converges toward it. This works brilliantly for the 80% of infrastructure work that's predictable and repeatable. But 20% of real infrastructure work isn't predictable. Interactive installers that ask questions nobody anticipated. Package upgrades that present merge conflicts in config files. Services that fail in ways that require investigation, not just a restart. Approval prompts that need human judgment. Diagnostics that require poking around, reading logs, trying things. wsh changes the equation. An AI agent can sit at every terminal simultaneously -- reading screens, handling prompts, making decisions. This enables an imperative, interactive model for the work that declarative tools can't handle. Not a replacement for Ansible. A complement for the cases where Ansible isn't enough. ## When to Use This **Use infrastructure-ops when:** - Operations require interactive input -- installers, prompts, approval dialogs, merge conflict resolution - Each host may behave differently and needs per-host judgment - You need to verify results interactively between steps (read logs, check endpoints, inspect state) - The operation is exploratory -- diagnosing an issue across a fleet, where what you do on host N depends on what you found on hosts 1 through N-1 - Rolling deployments need health verification with real traffic before proceeding to the next batch **Don't use infrastructure-ops when:** - The operation is fully predictable and idempotent -- use Ansible, Terraform, or your existing config management - You're provisioning infrastructure from scratch -- use Terraform or CloudFormation - The operation is a single command with no interaction -- use `parallel-ssh` or `ansible -m shell` - You don't need per-host decision-making -- fan out with a simple script instead ## Prerequisites You need a federated wsh cluster: a hub server with backends registered for each target machine. Each backend is a wsh server running on the target host. list servers # Expect: hub (local), plus one backend per target host If backends aren't registered yet, add them: add server at address http://10.0.1.10:8080 add server at address http://10.0.1.11:8080 add server at address http://10.0.1.12:8080 # Wait for all to become healthy loop: list servers if all target servers are healthy: break wait briefly retry See the **wsh:cluster-orchestration** skill for details on server registration, health monitoring, and authentication. You also need the patterns from **wsh:drive-process** (the send/wait/read loop) and **wsh:multi-session** (parallel session management, tagging, fan-out). This skill composes on top of both. ## Core Pattern: Same Operation Across N Hosts The fundamental infrastructure pattern: run the same operation on every host, but handle each host's response individually. This is different from `parallel-ssh` because you react to what each host does -- you don't just fire and forget. ### Sequential Fan-Out The safe default. Operate on one host at a time, verify success before moving to the next: hosts = ["web-1", "web-2", "web-3", "web-4", "web-5"] results = {} for host in hosts: create session "op-{host}" on server "{host}" send to "op-{host}": sudo systemctl restart myapp\n wait for idle on "op-{host}" read screen from "op-{host}" if password prompt detected: send to "op-{host}": {sudo_password}\n wait for idle on "op-{host}" read screen from "op-{host}" if success: results[host] = "ok" else: results[host] = "failed" # Decide: continue to next host, or stop? kill session "op-{host}" report results Sequential is slower but safer. If the operation breaks the first host, you haven't touched the other four. ### Parallel Fan-Out When the operation is safe to run everywhere simultaneously -- read-only commands, status checks, log collection: hosts = ["web-1", "web-2", "web-3", "web-4", "web-5"] # Launch all at once for host in hosts: create session "check-{host}" on server "{host}", tagged: fleet-check send to "check-{host}": systemctl status myapp\n # Poll for completion completed = {} while not all hosts completed: wait for idle on sessions tagged "fleet-check" (timeout 2000ms) # Returns whichever session settled first read screen from that session record result mark as completed # Use last_session + last_generation to advance # Clean up for host in hosts: kill session "check-{host}" report results ### Handling Per-Host Prompt Diversity The real reason this pattern exists. When you run `apt upgrade` across 10 servers, each one may present different prompts: - Server 1: no prompts, clean upgrade - Server 2: "Configuration file changed, keep or replace? [Y/n]" - Server 3: "Service nginx needs to be restarted. Restart now? [y/N]" - Server 4: "Pending kernel upgrade. Reboot required." - Server 5: dependency conflict requiring manual resolution A declarative tool must handle all these cases upfront with configuration flags. An agent reads each screen and responds with judgment: read screen from "upgrade-{host}" if "keep or replace" in screen: # Inspect the diff, decide based on whether we've # customized this config send: d\n # show diff first wait, read if config has local customizations: send: N\n # keep ours else: send: Y\n # take the new version if "restart now" in screen: # Check if this is a load-balanced service that can restart send: y\n if "reboot required" in screen: # Note for later -- don't reboot mid-upgrade-sweep results[host].needs_reboot = true This per-host judgment is the core value proposition. The agent adapts to what each host actually presents. ## Rolling Deployments Deploy to a fleet incrementally, verifying health between each batch. If something goes wrong, stop before it affects the whole fleet. ### The Rolling Deploy Pattern hosts = ["web-1", "web-2", "web-3", "web-4", "web-5", "web-6"] batch_size = 2 failed = [] for batch in chunk(hosts, batch_size): # Deploy this batch in parallel for host in batch: create session "deploy-{host}" on server "{host}", tagged: deploy send to "deploy-{host}": cd /opt/myapp && ./deploy.sh v2.1.0\n # Wait for all in this batch to finish for host in batch: wait for idle on "deploy-{host}" (timeout 30000ms) read screen from "deploy-{host}" if deployment prompt detected: handle prompt (see per-host prompt handling) wait for idle on "deploy-{host}" read screen from "deploy-{host}" if deployment failed: failed.append(host) # Health check this batch before proceeding for host in batch: create session "health-{host}" on server "{host}", tagged: health send to "health-{host}": curl -sf http://localhost:8080/health\n wait for idle on "health-{host}" read screen from "health-{host}" if health check failed: failed.append(host) kill session "health-{host}" # Clean up deploy sessions for this batch for host in batch: kill session "deploy-{host}" # Gate: stop if this batch had failures if failed is not empty: report: "Stopping rolling deploy. Failed hosts: {failed}" report: "Remaining hosts not deployed: {remaining}" break # Optional: wait for the new version to soak wait 30 seconds # Re-check health after soak period for host in batch: create session "soak-{host}" on server "{host}" send to "soak-{host}": curl -sf http://localhost:8080/health\n wait for idle, read screen if unhealthy: failed.append(host) kill session "soak-{host}" if failed is not empty: report: "Soak check failed. Stopping." break if failed is empty: report: "Rolling deploy complete. All hosts healthy." ### Choosing Batch Size - **Batch of 1**: Safest. Every host is individually verified. Slowest. Use for the first deploy of a risky change. - **Batch of N/3**: Good balance. You always have 2/3 of the fleet on the old version if something goes wrong. - **Batch of N-1**: Deploy to all but one (the canary). If the canary is still fine on the old version after you've deployed everywhere else, you know the issue is with the new version. ### Canary Pattern A special case of rolling deployment: deploy to one host first, soak for longer, then proceed if healthy: canary = hosts[0] rest = hosts[1:] # Deploy to canary create session "deploy-canary" on server "{canary}" send to "deploy-canary": ./deploy.sh v2.1.0\n wait for idle, read screen, handle prompts kill session "deploy-canary" # Extended health check on canary for i in range(5): wait 60 seconds create session "canary-check" on server "{canary}" send to "canary-check": curl -sf http://localhost:8080/health && echo "HEALTHY" || echo "UNHEALTHY"\n wait for idle, read screen if "UNHEALTHY" in screen: report: "Canary failed health check. Aborting." initiate rollback on canary stop kill session "canary-check" # Canary is healthy after 5 minutes. Deploy the rest. proceed with rolling deploy on rest ## Configuration Management Inspect current configuration, apply changes, and verify the result -- interactively, with the ability to inspect and judge at each step. ### Inspect-Modify-Verify The fundamental configuration pattern: for host in hosts: create session "config-{host}" on server "{host}" # 1. Inspect current state send to "config-{host}": cat /etc/myapp/config.yaml\n wait for idle, read screen record current_config[host] # 2. Decide whether to modify if config needs updating: # Back up first send: cp /etc/myapp/config.yaml /etc/myapp/config.yaml.bak\n wait for idle # Apply the change (using sed, patch, or writing a new file) send: sed -i 's/max_connections: 100/max_connections: 200/' /etc/myapp/config.yaml\n wait for idle # 3. Verify the change send: cat /etc/myapp/config.yaml\n wait for idle, read screen if change is correct: results[host] = "updated" else: # Restore backup send: cp /etc/myapp/config.yaml.bak /etc/myapp/config.yaml\n wait for idle results[host] = "reverted -- change didn't apply correctly" else: results[host] = "no change needed" kill session "config-{host}" ### Configuration Drift Detection Compare configurations across the fleet to find hosts that have drifted from the expected state: expected_config = "..." # known-good configuration drifted = [] for host in hosts: create session "audit-{host}" on server "{host}", tagged: audit send to "audit-{host}": cat /etc/myapp/config.yaml\n for host in hosts: wait for idle on "audit-{host}" read screen from "audit-{host}" if screen does not match expected_config: drifted.append(host) # Optionally: read scrollback for the full config if it's long kill session "audit-{host}" if drifted: report: "Configuration drift detected on: {drifted}" # Inspect each drifted host for details ### Interactive Configuration Tools Some services have their own interactive configuration tools -- `mysql_secure_installation`, `dpkg-reconfigure`, `certbot`, database migration wizards. These can't be driven by declarative tools, but an agent can handle them: create session "certbot-{host}" on server "{host}" send to "certbot-{host}": sudo certbot --nginx\n wait for idle, read screen # Certbot asks a series of questions if "Enter email address" in screen: send: admin@example.com\n wait for idle, read screen if "Terms of Service" in screen: send: A\n # Agree wait for idle, read screen if "Which names would you like to activate HTTPS for" in screen: send: 1\n # Select the first domain wait for idle, read screen # Verify the certificate was issued send: sudo certbot certificates\n wait for idle, read screen verify certificate is present and valid kill session "certbot-{host}" ## Package Management at Scale Package upgrades are one of the most common infrastructure tasks, and one of the most prone to interactive surprises. ### The Upgrade Pattern hosts = ["web-1", "web-2", "web-3", "db-1", "db-2"] for host in hosts: create session "upgrade-{host}" on server "{host}" # Update package lists send to "upgrade-{host}": sudo apt update\n wait for idle on "upgrade-{host}" (timeout 30000ms) read screen if password prompt: send: {sudo_password}\n wait for idle, read screen # Check what would be upgraded (dry run) send: apt list --upgradable\n wait for idle, read screen record pending_upgrades[host] # Review: show the operator what each host needs report pending upgrades per host # At this point, decide whether to proceed # Execute upgrades for host in hosts: send to "upgrade-{host}": sudo apt upgrade -y\n wait for idle (timeout 60000ms) read screen # Handle the prompts that -y doesn't suppress loop until shell prompt returns: if "Configuration file" in screen: # Package wants to overwrite a config file # Decide based on whether we maintain custom configs send: N\n # keep current version wait for idle, read screen if "Daemons using outdated libraries" in screen: send: \n # accept defaults wait for idle, read screen if "restart services" in screen: send: y\n wait for idle, read screen if shell prompt detected: break wait for idle (timeout 5000ms) read screen # Verify send: apt list --upgradable\n wait for idle, read screen if packages still pending: results[host] = "partial upgrade" else: results[host] = "fully upgraded" kill session "upgrade-{host}" ### Handling Package Manager Differences Different hosts may use different package managers. Detect and adapt: send to "detect-{host}": which apt yum dnf apk 2>/dev/null\n wait for idle, read screen if "apt" in screen: update_cmd = "sudo apt update && sudo apt upgrade -y" elif "dnf" in screen: update_cmd = "sudo dnf upgrade -y" elif "yum" in screen: update_cmd = "sudo yum update -y" elif "apk" in screen: update_cmd = "sudo apk update && sudo apk upgrade" send to "upgrade-{host}": {update_cmd}\n ### Specific Package Installation When you need to install a specific package with version pinning: for host in hosts: create session "install-{host}" on server "{host}", tagged: install send to "install-{host}": sudo apt install nginx=1.24.0-1~jammy\n wait for idle, read screen if "Do you want to continue?" in screen: send: Y\n wait for idle, read screen if "unable to locate package" in screen or "has no installation candidate" in screen: results[host] = "package not available" elif "is already the newest version" in screen: results[host] = "already installed" else: # Verify installation send: nginx -v\n wait for idle, read screen if "1.24.0" in screen: results[host] = "installed" else: results[host] = "unexpected version" kill session "install-{host}" ## Health Verification Post-operation checks are what separate a careful deployment from