| name | remote-dev |
| description | Operate the Mac mini remote dev host — connecting from the MacBook over mosh or herdr --remote, running and reattaching many Claude Code agents, herdr workspaces/panes and its socket API, claude --bg durable daemons, the Uptime Kuma readiness heartbeat, and the failure modes specific to a headless always-on Mac. Use when the user mentions herdr, mosh, the mini, "remote dev", "dev host", detaching or reattaching agents, "did my agent survive", sessions dying on lid-close, or asks how to reach a dev server running on the mini. |
Remote dev — the mini as dev host
The Mac mini is the always-on dev host; the MacBook is a thin client. Agents run
on the mini and outlive the MacBook.
Orientation first. Almost every mistake here comes from not knowing which
machine you are on. The SessionStart hook says: Backend: cache = the mini,
Backend: op = the MacBook.
Four layers — never collapse them
| Layer | Tool | Solves | Cannot solve |
|---|
| Reachability | Tailscale | stable address, NAT traversal | anything above it |
| Transport | mosh | keystrokes survive lid-close + roaming | multiplexing, port-forwarding, persistence |
| Persistence + UI | herdr (on the mini) | panes stay alive, per-pane agent state | anything about the network |
| Service exposure | Caddy | dev servers with real HTTPS + working WebSockets | anything about terminals |
Plus one thing that rides on top of all four and depends on none of them:
claude --bg.
Two different questions
Keep these apart — collapsing them is what makes this stack feel complicated:
| Question | Answer |
|---|
| How do I go look at the mini? | dev / desk — the transport layers below |
| How do I put work on the mini and check on it? | rd — no terminal needed at all |
Most days are the second one. The MacBook holds no repos any more, so rd
resolves every path on the host; that is why its commands take a repo name,
never a path.
repos [filter]
work <repo>
rd bg <repo> <task…>
agents
rd read <agent>
rd say <agent> "…"
work, agents, repos are shorthands; bg/read/say stay behind rd
because the bare names are a zsh builtin, a zsh builtin and /usr/bin/say.
rd routes itself off the secrets-backend marker — local exec on the mini, one
ssh hop from the MacBook — so the same words work on both machines.
Connecting from the MacBook
The normal way in is two shell functions (config/zsh/remote-dev.zsh):
dev [session]
desk [session]
Two mutually exclusive shapes. Persistence is identical either way — the
herdr server and its panes live on the mini regardless. This is purely a
client-experience choice.
mosh mini
herdr --remote mini
| transport | client on | roaming |
|---|
dev (mosh mini → herdr) | mosh/UDP | mini | survives lid-close with no reattach |
desk (herdr --remote mini) | ssh/TCP | MacBook | connection ends; re-run to reattach |
dev pins --experimental-remote-ip=remote — load-bearing, not cosmetic. mosh's
default proxy mode passes -S none to ssh (disabling multiplexing), so every
launch opened a fresh connection and popped its own 1Password biometric
approval — exactly the friction the ControlMaster block in ssh_config
exists to remove. remote mode reuses the master.
--remote at the desk, mosh on the road. mosh does not forward the SSH
agent (upstream refuses it), nor port forwarding, OSC 52 clipboard, or
sixel/kitty graphics — and its predictive echo only engages on a laggy link, so
over a LAN-latency tailnet hop it buys nothing ssh wasn't already doing. Its
value is specific and real: a bad link, and roaming without reattaching.
herdr attach is not a command — the real forms are herdr session attach <name>
and herdr agent attach <target>.
mosh needs the ACL's udp:60000-61000 grant. Without it the ssh handshake
succeeds and the session then hangs forever — it reads as a broken mosh, not a
blocked port.
herdr day to day
herdr
herdr --session <name>
herdr session list|attach|stop
herdr status
Server lifecycle is brew services, not herdr server stop — it runs with
KeepAlive, so a bare stop just gets relaunched (same as colima):
brew services restart herdr
The socket API is the agent-facing door
Every pane is scriptable without attaching a TUI, which is what makes herdr
usable from an agent rather than only by a human:
herdr workspace list|create|focus|close
herdr pane list|current|zoom
herdr agent list
herdr agent read <target> [--source visible|recent|recent-unwrapped|detection]
herdr agent prompt <target> "<text>" [--wait]
herdr agent wait <target> --until <status>
herdr agent start <name> --kind claude --pane <id>
herdr workspace create works with zero clients attached — the server is
genuinely headless.
Agent status needs the integration hook
make herdr-setup installs herdr's first-party Claude Code integration. Without
it every pane reports agent_status: "unknown", which throws away the one thing
herdr has over tmux. It is safe on any machine — the hook exits 0 immediately
unless HERDR_ENV / HERDR_SOCKET_PATH / HERDR_PANE_ID are set.
make herdr-setup is opt-in and not part of the setup chain. It has not been
run on the MacBook, which has the guarded settings entry but no
~/.claude/hooks/herdr-agent-state.sh — that is exactly what the test -f guard
below covers. Run it there too if you want pane status from a MacBook-side client.
The settings entry is tracked in dotfiles/config/settings.template.json,
because make setup merges settings.json with the template winning on
hooks — an entry added only by herdr integration install claude gets
deleted on the next make setup. If pane status ever goes back to unknown,
that is the first thing to check.
Durable agents — claude --bg
rd bg <repo> '<task>'
claude agents --json
claude attach|logs|stop <id>
--bg reparents to PID 1 as claude daemon run, so it survives ssh death,
herdr death and lid-close independently of every layer above.
It survives independently; it cannot be launched independently
--bg is freestanding once running, which is not the same as freestanding at
spawn time — and the difference fails silently.
Never ssh mini 'claude --bg …' directly. Claude Code's Max credentials
live in the login keychain, which an ssh session cannot reach. The daemon
starts anyway, prints Not logged in · Please run /login, falls back to
API Usage Billing, and still looks healthy in claude agents. You get a
running agent that is off Max and doing nothing.
The herdr server is a brew service under launchd inside the user's GUI
session, so anything it spawns inherits keychain access. rd bg therefore
launches through a throwaway herdr pane and closes it once the daemon exists.
Verified both directions 2026-07-27: identical command, Not logged in over
ssh, Claude Max through a pane.
The same reasoning covers anything else on the mini that needs the login
keychain — a LaunchAgent, a cron line, a daemon started from a script. Being on
the tailnet is not the same as being in the GUI session.
This matters more than it looks. A kill -9 of the herdr server brings the
workspace back by name but with a new terminal_id — the layout is restored and
every process inside it is gone. herdr is pre-1.0. So:
Anything that must not die goes in claude --bg, not in a herdr pane.
An interactive session is exactly the kind that dies with its connection. If the
user says "my agent should keep running", check claude agents --json for
kind: interactive before assuming it will.
Reaching a dev server on the mini
A dev server bound to 127.0.0.1:PORT on the mini is reachable from any Mac on
the tailnet as https://<mini-magicdns>:PORT, with a real cert.
vim ~/.config/caddy-tailnet.ports
make caddy-tailnet
Do not use tailscale serve for dev servers — issue #18827 drops WebSockets
every 10-40s, which is HMR breaking on a timer. serve stays right for the
always-on rows (:7730 rb, :8443 IU dashboard Funnel).
Three failure modes, all silent:
| Symptom | Cause |
|---|
| Connection times out, nothing in any log | The Tailscale ACL has no grant for that port. tag:mac → tag:mac covers tcp:22, tcp:5900, tcp:7700-7799, udp:60000-61000 — anything else needs adding, exactly as rb's :7730 did |
| Caddy won't start, or the dev server can't bind its port | The generated block is missing bind <tailnet-ip>, so Caddy took 0.0.0.0:PORT and collided with the dev server on 127.0.0.1:PORT |
| 403 from the app itself | Vite 5.4.12+ DNS-rebinding guard. Add the MagicDNS host to server.allowedHosts (rb does a .ts.net suffix match) |
Changing the ACL needs both machines: the repo is on the mini, but
tailscale-acl-push needs the Tailscale API key, which is op://Private/* and
refused by the mini's cache by design. Edit on the mini, push from the MacBook.
Monitoring
One composite heartbeat (herdr + sshd + tailscaled + mosh) pushes to the
MacMini Dev Host - Push Uptime Kuma monitor every 5 minutes.
make devhost-health-check
make devhost-health-setup
tail ~/Library/Logs/devhost-health.log
Push, not probe: the ACL grants tag:homelab → tag:vps but not
tag:homelab → tag:mac, so Uptime Kuma cannot reach the mini at all. The mini
reports on itself over the already-granted outbound path.
make remote-dev-doctor (scripts/remote-dev-doctor.sh) is the MacBook-side
counterpart — it verifies the path FROM the MacBook, which the mini-side
heartbeat structurally cannot: the mini holds no key material and cannot ssh to
itself, so it can't see inbound auth, ControlMaster reuse, agent forwarding,
or the mosh UDP path. Read-only, currently 10/10 passing: tailscale reachability
(direct vs DERP), ssh, ControlMaster, agent forwarding, mosh installed locally,
mosh-server on the mini's non-interactive PATH, mosh-server in the mini's
Application Firewall allowlist, herdr server running, the agent-state hook
present, and a resolvable GitHub credential.
Failure modes specific to this host
| Symptom | Cause |
|---|
mosh mini hangs after a clean ssh handshake | ACL missing udp:60000-61000. Verified present 2026-07-26 — check the Application Firewall row below first, it presents identically |
herdr --remote mini asks "restart the remote server now? [y/N]", warning it "may not survive SSH connection loss" | Answer N. False positive for this host: the server is the brew service, PID owned by launchd (PPID 1), so no ssh disconnect can reach it — verified continuously up across a whole session of opening and closing connections. Answering y restarts it outside brew services supervision and kills every process in every pane |
mosh prints Error: vector and exits | The client's terminal has no window size (a 0×0 pty). Real terminals are fine; this bites scripted/automated launches — set TIOCSWINSZ before exec |
| Doubled keystrokes over ssh | TERM=xterm-ghostty reaching a host without that terminfo (cmux #2969). ssh_config pins SetEnv TERM=xterm-256color |
launchctl print … sshd says state = not running | socket-activation idle, not a fault. Check netstat -an | grep '\.22 .*LISTEN' |
ssh localhost fails on the mini | By design — the mini holds no private key material. Inbound auth is the connecting machine's key, so the mini cannot test its own inbound ssh. Verify from the MacBook |
A direct op read / op run hangs on the mini | No biometric prompt to answer. Use secrets-run. op whoami fails fast, so preflight guards are safe |
op signin "worked" but the next command says not signed in | The session lives in the shell that ran it. Chain them: op signin --account tkrumm && <cmd> |
| Agent died when the lid closed | It was kind: interactive. Use rd bg |
A --bg agent runs but does nothing; claude logs shows Not logged in · Please run /login and the banner says API Usage Billing instead of Claude Max | It was spawned over ssh, which cannot reach the login keychain. Spawn through a herdr pane — that is exactly what rd bg does. Never ssh mini 'claude --bg …' |
rd says "herdr server is not running" | brew services restart herdr on the mini; make remote-dev-doctor to confirm the rest of the path |
| Workspace came back but the work is gone | herdr server restarted — layout persists, processes do not |
claude: command not found over non-interactive ssh to the mini | Fixed — make setup's _setup-zshenv block now puts both Homebrew and ~/.local/bin on the non-interactive PATH. If it recurs, that block is missing: re-run make setup on the mini |
| mosh connects over ssh then hangs / "did not make a successful connection to <ip>:<port>" | mosh-server missing from the mini's Application Firewall allowlist. Fix: make mosh-firewall. Re-run after brew upgrade mosh |
git push on the mini fails with could not read Username for 'https://github.com' | The credential helper returned nothing. Fix: make git-headless, and reseed with make secrets-seed if the cache is stale. The helper exits 0 with an empty body when the token is unresolvable, which is why the failure looks like a transport fault |
| Any remote-dev symptom you can't immediately place | Run make remote-dev-doctor first — it names the failing layer instead of guessing |
Rules
- Manage the herdr server with
brew services, never a bare herdr server stop.
- Never run
herdr update. It downloads and installs over the
brew-managed binary, so the version stops matching the Brewfile — the repo's
supply-chain audit trail — and make brew-check starts reporting drift.
Upgrade with brew upgrade herdr, deliberately, one package at a time.
herdr is pre-1.0 and ships often; that cadence is a reason to gate upgrades
through the manifest, not to bypass it.
- Never run a validation loop or an edit over files a running
@implementer
subagent owns — that applies here too, across panes.
- Do not add
tailscale serve bindings for dev servers; use Caddy. Tailscale
issue #18827 drops WebSockets through serve/funnel every 10-40s, which is HMR
breaking on a timer.
- The
:8443 Funnel row is the IU dashboard, public by design, gated by
tag:iu-dashboard-funnel — an additive single-device tag that keeps Funnel
capability off the work MacBook. Do not "clean it up".
Full model and rationale: dotfiles/docs/remote-dev.md.
Access/auth model: dotfiles-private/docs/access-model.md.
