cluster-orchestration

> **IMPORTANT: EXECUTION CONTEXT** > This skill describes *what to do* — domain patterns and decision-making. > It does NOT describe *how* to call the API. > > 1. **If you have `wsh_*` tools** (check your toolkit for `wsh_send_input`, > `wsh_get_screen`, etc.): use them directly. Operation names in this > skill generally map to tool names (e.g., "send input" → `wsh_send_input`). > When in doubt, list your available `wsh_*` tools. > 2. **If you do NOT have `wsh_*` tools**: you are in HTTP/curl fallback mode. > **DO NOT GUESS endpoints or CLI subcommands.** > Load the full API reference first: search your workspace for > `skills/core/` and read `SKILL.md`. It contains every endpoint > with working curl examples and a bootstrap sequence. > 3. **Quick bootstrap**: `curl -sf --unix-socket ${XDG_RUNTIME_DIR:-/tmp}/wsh/default.http.sock http://localhost/health` > — if that fails: `wsh server -L agent-$$ --ephemeral &` and retry. # wsh:cluster-orchestration — Distributed Terminal Sessions Sometimes one machine isn't enough. You need to spread work across multiple servers — running builds on beefy hardware, tests in isolated environments, deployments on production hosts. Cluster orchestration lets you manage sessions across a fleet of wsh servers from a single hub. ## When to Use Cluster Orchestration **Use cluster orchestration when:** - Work needs to run on specific machines (different hardware, different networks, different environments) - You need to scale beyond what one machine can handle - Sessions need isolation across physical or virtual boundaries - You're coordinating a distributed workflow (build here, test there, deploy somewhere else) **Don't use cluster orchestration when:** - All your work fits on one machine — use multi-session instead - You only need one session — use the basic session primitives - The tasks don't benefit from distribution ## Concepts ### Hub and Backends A cluster has one **hub** server and one or more **backend** servers. The hub is the server you talk to — it receives your requests and either handles them locally or forwards them to the right backend. Backends are regular wsh servers that the hub knows about. You interact exclusively with the hub. The hub handles routing, health monitoring, and aggregation transparently. From your perspective, it looks like one server with sessions spread across multiple machines. ### Server Identity Every server in the cluster has a **hostname** — a unique identifier. Backends acquire their hostname automatically when they connect to the hub (the hub queries each backend's identity). The hub's own hostname is its system hostname or a configured override. You use hostnames to target specific servers when creating sessions or querying state. ### Health Monitoring The hub continuously monitors each backend's health. A backend can be in one of three states: - **healthy** — connected and responding normally - **connecting** — initial connection in progress or reconnecting after a disruption - **unavailable** — connection lost, not responding Only healthy backends participate in session operations. The hub automatically reconnects to backends that become unavailable, so transient network issues resolve on their own. ## Server Registration and Monitoring ### Discovering the Cluster Before creating sessions on remote servers, check what's available: list servers This returns every server in the cluster, including the hub itself. Each entry shows the hostname, health status, and role. The hub always appears as "local" with health "healthy". ### Adding a Backend Register a new backend server with the hub. Backend addresses require a scheme (`http://` or `https://`) and may include a path prefix for reverse-proxy deployments: add server at address http://10.0.1.10:8080 add server at address https://proxy.example.com/wsh-node-1 The hub immediately begins connecting to the backend. It starts in "connecting" state and transitions to "healthy" once the connection is established and the backend's hostname is resolved. If the backend requires authentication, provide a token: add server at address http://10.0.1.10:8080 with token "secret" ### Checking a Specific Server Get detailed status for a single server by hostname: get server "prod-1" ### Removing a Backend Deregister a backend when it's no longer needed: remove server "prod-1" This disconnects from the backend and removes it from the cluster. Sessions that were running on that backend become inaccessible through the hub (they continue running on the backend itself). ### Waiting for Backends to Become Healthy After adding a backend, you need to wait for it to become healthy before creating sessions on it. Poll the server list until the backend's health transitions from "connecting" to "healthy": loop: list servers if target server is healthy: break wait briefly retry This typically takes a few seconds. Don't proceed with session creation until the backend is healthy — requests to unhealthy backends will fail. ## Distributed Session Creation ### Creating a Session on a Specific Backend Target a specific server by hostname when creating a session: create session "build" on server "prod-1" The hub forwards the creation request to the named backend. The session runs on that backend's hardware, in its environment, with its resources. All subsequent operations on that session are automatically routed through the hub to the right backend. ### Creating Local Sessions Sessions created without a server target run on the hub itself: create session "local-work" This is exactly the same as single-server operation. The hub handles it locally without involving any backend. ### Choosing Where to Place Work Consider these factors when deciding where to create sessions: - **Hardware requirements**: CPU-intensive builds on powerful machines, memory-heavy tests on high-RAM servers - **Network locality**: Operations that access local resources (databases, filesystems) should run on the same machine - **Isolation**: Untrusted or experimental work on dedicated backends, away from production sessions - **Load distribution**: Spread parallel work across backends to avoid overloading any single machine ## Tag-Based Cross-Server Workflows Tags work transparently across server boundaries. Sessions on different backends can share the same tags, and tag-based queries aggregate results from all healthy servers. ### Distributed Fan-Out Spread parallel work across the cluster using tags to track it: create session "build-api" on server "build-1", tagged: ci create session "build-web" on server "build-2", tagged: ci create session "test-e2e" on server "test-1", tagged: ci send each session its respective command wait for idle across sessions tagged "ci" The idle detection races across all tagged sessions regardless of which server they're on. The first to settle is returned. ### Listing Sessions Across Servers list sessions Without a server filter, the session list aggregates across all healthy backends plus the hub. Each session in the response includes a `server` field indicating which server it lives on. list sessions tagged "ci" Tag filtering also works across the full cluster. Only sessions matching the tag are returned, from any server. list sessions on server "build-1" To see sessions on a specific backend only, filter by server. ### Session Operations Are Transparent Once a session exists, all operations work the same regardless of where it lives. The hub routes requests automatically: send input to "build-api": cargo build\n wait for idle on "build-api" read screen from "build-api" kill session "build-api" You don't need to remember which server a session is on. The hub tracks this mapping and routes transparently. ## Cross-Server Quiescence Patterns ### Waiting for Any Session to Settle The server-level idle detection races across all sessions in the cluster: wait for idle across all sessions (timeout 2000ms) Returns whichever session becomes idle first, including its name and the server it's running on. Use `last_session` and `last_generation` to avoid re-returning the same session. ### Waiting for a Tagged Subset wait for idle across sessions tagged "build" (timeout 2000ms) This is the most common pattern for distributed fan-out. Tag all related work, then poll idle across the group. ### Polling a Specific Backend's Sessions If you need to check just one server's sessions: list sessions on server "build-1" for each session: wait for idle read screen check results ### Coordinating Sequential Stages When one stage must complete before the next begins: # Stage 1: Build on the build server create session "build" on server "build-1", tagged: pipeline send to "build": make release\n wait for idle on "build" (timeout 5000ms) read screen from "build" # verify success # Stage 2: Test on the test server create session "test" on server "test-1", tagged: pipeline send to "test": ./run-tests.sh\n wait for idle on "test" (timeout 5000ms) read screen from "test" # verify success # Stage 3: Deploy on the deploy server create session "deploy" on server "deploy-1", tagged: pipeline send to "deploy": ./deploy.sh\n ... Each stage runs on a different server but follows the same send/wait/read/decide loop. ## Failure Handling ### Backend Goes Down When a backend becomes unavailable: - **Existing sessions on that backend become inaccessible.** Operations targeting those sessions will fail with a server unavailable error. - **The hub continues operating normally.** Local sessions and sessions on other healthy backends are unaffected. - **The hub automatically attempts to reconnect.** If the backend comes back, the connection is re-established and its sessions become accessible again. - **Session listing excludes unavailable backends.** Only sessions from healthy servers appear in aggregated listings. ### Recovery Strategies **Check health before critical operations:** list servers if target server is unavailable: fall back to another server or report the failure **Design for partial failure:** When fanning out across multiple backends, some may fail while others succeed. Collect results from the successful ones and handle failures individually rather than treating any failure as a total failure. results = {} for each session in the fan-out: try: wait for idle read screen results[session] = success catch server unavailable: results[session] = failed report partial results **Use tags for recovery:** If a backend fails mid-workflow, you can recreate the affected sessions on a different backend with the same tags: # Original session on failed backend # create session "build" on server "build-1", tagged: ci # Recovery: recreate on another backend create session "build-retry" on server "build-2", tagged: ci send to "build-retry": (same command) ### Session Lifetime and Server Lifetime Sessions are owned by the backend they run on. If you remove a backend from the cluster, its sessions continue running on that machine — they just become unreachable through the hub. If the backend process exits, its sessions end. The hub doesn't migrate sessions. If a backend goes down and its sessions are lost, you need to recreate them on another backend. ## Pitfalls ### Don't Over-Distribute Distribution adds latency and complexity. Every request to a remote session goes through the hub to the backend and back. If all your work can run on one machine, use multi-session on a single server. ### Monitor Backend Health Don't assume backends are always available. Check health before starting critical workflows, and design for graceful degradation when backends fail. ### Clean Up Remote Sessions Remote sessions consume resources on backend machines. Clean up after yourself — don't leave orphaned sessions running on backends. The hub won't automatically kill sessions when you remove a backend. ### Backend Authentication Backends may require authentication tokens. Ensure tokens are configured correctly when adding backends. Without proper authentication, the hub won't be able to connect. ### IP Access Control When the hub has an `[ip_access]` section in its federation config, backend addresses are checked against the blocklist and allowlist at registration time. Backends whose resolved IPs fall outside the allowed ranges will be rejected. There is no hardcoded blocklist -- the operator owns the threat model. ### Hostname Uniqueness Every server in the cluster must have a unique hostname. If two backends have the same hostname, the second registration will be rejected. Configure unique hostnames for each backend if the system hostnames collide.