fleet

name	fleet
description	Fleet operations — use when a task involves other machines (discovering, reaching, or reasoning about them), installing/updating/removing any user-level tool, converging a machine, or working with paseo or the agent-fleet repo.

Fleet Operations

The fleet is defined by the agent-fleet repo (~/agent-fleet on every managed machine). One exact-pinned pixi-global manifest (base + per-machine overlay) declares each machine's tools; a machine converges to it — the manifest is the complete state, and converging installs what's missing AND deletes what's unlisted.

Discovering machines — query, never hardcode

Question	Source of truth	Command
What exists / is online?	Tailscale (self-updating)	tailscale status
Specs, GPUs, roles?	Rackpeek (curated, can lag)	ssh pablo-dl-server docker exec rackpeek rpk summary (also rpk servers describe <name>, rpk systems list)
Fleet-managed?	agent-fleet repo	overlay at manifests/machines/<lowercased short hostname>.toml (the LOCAL hostname, which may differ from the Tailscale name — e.g. the M5 laptop is pablos-m5-pro on the tailnet but macbook-pro-3.toml here, and the DGX is dgx-spark vs spark-9232.toml); ~/agent-fleet checked out

Stable facts: pablo-dl-server is the conductor (always-on, hosts rackpeek). The DGX Spark (dgx-spark on the tailnet, local hostname spark-9232) is linux-aarch64 — account for it in platform availability. Macs are osx-arm64.

Reaching machines

SSH rides the tailnet: any hostname from tailscale status is an SSH target (ssh pablo-ubuntu, ssh dgx-spark; ~/.ssh/config may hold shorter aliases). Patterns that matter:

• Non-interactive commands: ssh -o ConnectTimeout=5 -o BatchMode=yes <host> '…' — BatchMode fails fast instead of hanging on a password prompt.
• PATH gotcha: non-interactive SSH does NOT load the shell rc, so pixi tools are missing. Prefix every remote command chain with export PATH="$HOME/.pixi/bin:$PATH".
• pkill -f <pattern> over SSH can match and kill your own SSH session's command line — prefer pgrep -a first, then kill exact PIDs.

Moving files — scripted transfers always use SSH; Taildrop is for humans:

• Single file, exact destination: scp file <host>:path.
• Directory trees / preserving structure: COPYFILE_DISABLE=1 tar czf - . | ssh <host> 'tar xzf - -C <dest>' (COPYFILE_DISABLE strips macOS xattr noise).
• Phone/iPad ↔ machine: Taildrop (share sheet / tailscale file cp) — its niche is devices without SSH. No destination control: macOS receives to ~/Downloads; headless Linux needs a second step (tailscale file get). Alpha; requires the tailnet "Send Files" setting.

Tailscale itself is OS-layer (never pixi-managed); the tailnet is also the auth for Tailscale SSH on some machines (e.g. spark).

Changing a machine's tools

1. Edit manifests/base.toml (fleet-wide) or manifests/machines/<host>.toml (one machine). Exact == pins; base packages must solve on linux-64, linux-aarch64, AND osx-arm64. Channels: conda-forge → own channels (prefix.dev/ai-demos, prefix.dev/my-skill-forge) → prefix.dev/github-releases as gap-filler.
2. pixi run check, commit, push.
3. Converge each machine: ~/agent-fleet/scripts/sync.sh.

Done when every target machine prints ✓ <host> converged to <rev> at the pushed revision — not before.

JS CLIs get rattler-build recipes in the ai-demos repo (npm-tarball pattern, see recipes/ruler there) and arrive via the channel.

Adding a skill fleet-wide

Skills are agent-skill-<name> conda packages from https://prefix.dev/my-skill-forge. Three steps, each explicit:

1. Recipe in the my-skill-forge repo (vendored SKILL.md, noarch: generic, agentskills validate in tests — copy an existing recipe). Merging publishes to the channel.
2. Pin it in the fleet: add agent-skill-<name> = "==<version>" under [envs.agent-skill-forge.dependencies] in agent-fleet's manifests/base.toml, push.
3. Converge machines — link_skills in sync.sh symlinks it into ~/.claude/skills and ~/.codex/skills (work profile shares via ~/.claude-work/skills).

Publishing alone does NOT propagate: the pin is the membership act (weekly pin-bump only bumps versions of skills already pinned — it never adds new ones). Done when the skill dir appears under ~/.claude/skills/ on every converged machine.

MCP servers (fleet-managed since 2026-07-07)

ONE declared MCP set in git lands on claude (personal), claude-work, AND codex on every machine: [mcp_servers.<name>] blocks in agent-fleet's .ruler/ruler.toml → pixi run render (committed .mcp.json) → converge (scripts/mcp_apply.py applies at user scope through each CLI's own mcp add/remove — it never writes their state files directly).

• NEVER hand-run claude mcp add / codex mcp add for a fleet server — hand edits are drift and revert on the next converge. Hand-added servers under OTHER names are ignored (the applier only touches declared names).
• Entries persist the machine's RESOLVED binary path (a capability probe vouches for the exact binary sessions will run); too-old binaries are skipped with a mcp: skip <name> … note and apply automatically once the pin catches up (e.g. rerun viewer-mcp needs >= 0.34).
• New sessions pick changes up automatically; running sessions restart.

Holds

• Range pin (tmux = "3.3.*"): the range IS the hold — pin-bump ignores it.
• # pin-bump: hold on an == pin line: frozen exact pin.

Updates

Mon 05:00 UTC: ai-demos autobump PRs recipe bumps (auto-merges on green CI, publishes). Mon 06:00: agent-fleet pin-bump PRs base.toml updates — human-gated; merging converges the fleet on next sync. Rollback = revert the pin PR.

Paseo (orchestration layer)

Fleet-managed on every machine (2026-07-06): CLI daemon from the pixi paseo pin, run by a service unit — systemctl --user status|restart paseo on Linux (linger required for boot start), launchctl print|kickstart -k gui/$UID/sh.paseo.daemon on macOS (starts at LOGIN). NEVER paseo daemon start by hand — the unit owns the daemon. paseo daemon status for health.

Config: ONE file in git, agent-fleet/config/paseo/config.json — edit, push, converge (scripts/paseo_apply.py seds in the machine's tailnet IP and restarts only on fleet-initiated change via the .fleet-config-applied shadow copy; UI edits are drift that reverts on the next fleet change). State lives in ~/.paseo (machine-local, survives binary swaps). There is NO daemon password — auth is the daemon keypair + relay pairing. One web UI: https://pablo-dl-server.ilish-ruler.ts.net:8767 (PWA); pair a daemon into it with paseo daemon pair → Add connection.

When a converge fails or a machine misbehaves

Read references/internals.md — sync.sh's pipeline and safety gates, capture/overlay onboarding, and known failure signatures (dead OAuth masquerading as "model unavailable", stale stamps, app configs broken by version bumps).