tailscale-mosh-recover

name	tailscale-mosh-recover
description	Use when mosh hangs / fails to connect to a headless Mac mini after a Tailscale update or restart, OR when a remote-access tool (RustDesk, VNC, AnyDesk) fails to reach its public relay from an MDM-managed client. Layers — Tailscale daemon health (dual-install conflict, headless GUI-app limitation, Homebrew launchd takeover), downstream mosh-server cleanup (stale UDP bindings to old Tailscale IP), macOS resolver stuck state, and Tailscale direct-IP as relay-bypass fallback. Symptoms include `could not get canonical name for <host>`, `failed to connect to local Tailscale service`, `Tailscale.CLIError error 1`, `Connection closed by UNKNOWN port 65535`, `Failed to connect to rs-ny.rustdesk.com:21116: Please try later`.
allowed-tools	["Read","Glob","Grep","Bash(tailscale:*)","Bash(brew:*)","Bash(ps:*)","Bash(lsof:*)","Bash(kill:*)","Bash(tmux:*)","Bash(which:*)","Bash(ls:*)","Bash(ln:*)","Bash(rm:*)","Bash(sudo:*)","Bash(hash:*)","Bash(mdls:*)","AskUserQuestion"]

Tailscale + Mosh Recovery (Headless Mac mini)

Problem

After a Tailscale update or daemon restart on a headless Mac mini, mosh sessions break. Common symptoms (any combination):

• tailscale status returns: failed to connect to local Tailscale service; is Tailscale running?
• mosh <host> returns: could not get canonical name for <host>: nodename nor servname provided, or not known, then Connection closed by UNKNOWN port 65535, then Did not find remote IP address (is SSH ProxyCommand disabled?).
• Tailscale GUI app CLI returns: Tailscale.CLIError error 1.
• which tailscale reports a path that does not actually exist on disk.
• tailscale up registers the machine as <hostname>-1 (a duplicate of the original node — IP changed).

The two underlying problems are causally linked: Tailscale daemon trouble forces a re-auth → new Tailscale IP → existing mosh-server processes are bound to the old IP and become unreachable.

Context / Triggers

Sign	Likely cause
Mac mini's tailscale status says daemon not running	GUI app daemon needs a logged-in user session — won't run headless
which tailscale returns a path that ls says doesn't exist	Shell PATH cache + missing symlink (often the Homebrew symlink dropped during update)
Two Tailscale installs: GUI app at /Applications/Tailscale.app AND Homebrew Cellar	Dual install — one's daemon is fighting for the socket
mosh fails only after a working tailscale up re-registration	Stale mosh-server processes bound to the prior Tailscale IP
Old hostname offline + new hostname <name>-1 in admin console	Re-auth created a new node; old one is stale

Solution

Work in this order: Layer 1 (Tailscale daemon) then Layer 2 (mosh cleanup). If only mosh is broken and tailscale status is healthy, skip Layer 1.

Layer 1: Tailscale daemon health (headless Mac mini)

The headless constraint: the standalone Tailscale.app GUI bundle's daemon needs an active user session. On a headless Mac mini (no monitor, no auto-login), the GUI daemon never starts. The right setup for headless is Homebrew's tailscale package run as a launchd service — it's a system service that survives reboots without any logged-in user.

Step 1.1: Confirm the symptom + identify installs

tailscale status              # likely: "failed to connect to local Tailscale service"
which tailscale               # may report a path that doesn't exist
ls -la /Applications/Tailscale.app  # GUI app present?
brew list tailscale 2>/dev/null     # Homebrew install present?
ls -la $(brew --prefix)/Cellar/tailscale  # Cellar contents if Homebrew has it

If both installs exist, that's the dual-install conflict. The CLI symlink at /opt/homebrew/bin/tailscale may have been removed by a recent update — leaving the daemon running (or not) and no CLI to talk to it.

Step 1.2: Remove the GUI app's broken CLI shim (if present)

The GUI app installs a shim at /usr/local/bin/tailscale that errors with Tailscale.CLIError error 1 when the GUI app isn't running. Remove it to avoid PATH ambiguity:

sudo rm -f /usr/local/bin/tailscale

Step 1.3: Re-link the Homebrew CLI

The Homebrew binary still exists in $(brew --prefix)/Cellar/tailscale/<version>/bin/tailscale; it's just unlinked.

brew link tailscale
hash -r                       # clear shell PATH cache
which tailscale               # should now report /opt/homebrew/bin/tailscale

If brew link complains about a conflict, the conflicting path was likely the GUI shim from Step 1.2 — re-run Step 1.2 and try again. If you've decided to remove the GUI app entirely, brew link --overwrite tailscale is acceptable.

Variant: orphan-after-autoremove

If which tailscale returns nothing right after a brew uninstall of something else reported Autoremoving ... tailscale: the daemon is still running on the unlinked binary — brew link --overwrite tailscale restores the CLI without touching it. Full diagnosis + 2026-05-22 incident: references/extended-diagnostics.md §Variant.

Step 1.4: Confirm the Homebrew daemon is running as a system service

sudo brew services list | grep tailscale

Expected: tailscale started root /Library/LaunchDaemons/homebrew.mxcl.tailscale.plist.

If started and root are correct, the daemon is already healthy — skip to Step 1.5. If not started:

sudo brew services start tailscale

The sudo matters — without it, tailscaled runs as your user and lacks the privileges for subnet routing / exit node mode.

Step 1.5: Re-authenticate

tailscale up

Follow the auth URL on another device. If the prior config used flags (exit node, subnet routes, etc.), include them — for example:

sudo tailscale up --advertise-exit-node --advertise-routes=192.168.1.0/24 --accept-routes

Re-auth creates a fresh node identity, which is why the machine's Tailscale IP can change. Note the new IP — Layer 2 depends on it.

tailscale status              # confirm: tailnet visible, node "active"

Step 1.6: Clean up the GUI app (recommended for headless)

For a headless Mac mini, the GUI app is dead weight and risks dropping a broken CLI shim back on future updates:

sudo rm -rf /Applications/Tailscale.app
sudo rm -f /usr/local/bin/tailscale
# Optional: user-level support files (vary by App Store vs standalone build):
rm -rf ~/Library/Containers/io.tailscale.ipn.macos
rm -rf ~/Library/Group\ Containers/*.io.tailscale.ipn.macsys
rm -rf ~/Library/Application\ Support/Tailscale
rm -rf ~/Library/Preferences/io.tailscale.ipn.macos.plist
rm -rf ~/Library/Caches/io.tailscale.ipn.macos
# Verify nothing GUI-related in LaunchAgents (Homebrew daemon plist must remain):
ls -la /Library/LaunchDaemons/ | grep -i tailscale  # keep homebrew.mxcl.tailscale.plist
ls -la /Library/LaunchAgents/ ~/Library/LaunchAgents/ 2>/dev/null | grep -i tailscale

Step 1.7: Admin-console cleanup (browser, not CLI)

Visit https://login.tailscale.com/admin/machines:

• Delete the old offline node (the one with the original hostname before re-auth).
• Rename the new node back to the clean hostname if you ended up with <name>-1.
• Re-approve advertised routes / exit-node status — these need explicit approval per node.

Layer 2: Mosh-server cleanup (downstream of any Tailscale IP change)

After any Tailscale re-auth or daemon restart that changes the machine's IP, existing mosh-server processes are bound to the old IP and become unreachable. tmux sessions remain alive and untouched — only the mosh-server wrapper needs replacing.

Step 2.1: List mosh-servers and their UDP bindings

ps aux | grep mosh-server | grep -v grep
lsof -nP -iUDP 2>/dev/null | grep mosh

The lsof output shows each mosh-server's PID and bound IP/port (e.g. mosh-serv 94933 ... UDP 100.x.y.z:60005). Compare the bound IPs against the current Tailscale IP from tailscale status (e.g. 100.x.y.z). Any mosh-server bound to a different IP is stale.

Step 2.2: Verify which tmux sessions correspond to which mosh-servers

tmux ls
ps -ef | grep mosh-server | grep -v grep   # `mosh-server new ... -- tmux new -s <SESSION>` shows the session each was attached to

This lets you confirm a stale mosh-server is the wrapper for a specific tmux session — that session's work is preserved on disk by tmux regardless of mosh-server fate.

Step 2.3: Kill stale mosh-servers (preserve the healthy one)

kill <PID-list-of-stale-servers>

Do not include any mosh-server whose UDP binding matches the current Tailscale IP. tmux sessions survive.

Step 2.4: Verify

ps aux | grep mosh-server | grep -v grep   # only the healthy ones remain
lsof -nP -iUDP | grep mosh                 # bindings should all show the current IP
tmux ls                                    # sessions unchanged

Step 2.5: Reconnect from the remote machine

From the client (e.g. MacBook):

mosh <host> -- tmux a -t <session-name>

This spawns a fresh mosh-server bound to the current Tailscale IP and reattaches to the existing tmux session. If mosh <host> fails with the canonical-name error, fall back to the resolved hostname directly:

mosh <my-mini-or-actual-tailscale-name> -- tmux a -t <session-name>

Verification

After Layer 1 + Layer 2:

# On the Mac mini:
which tailscale                            # /opt/homebrew/bin/tailscale
tailscale status                           # shows tailnet; this node "active"
sudo brew services list | grep tailscale   # tailscale started root
ps aux | grep mosh-server | grep -v grep   # only mosh-servers bound to current IP

# From the client:
mosh <host> -- tmux a -t <session-name>    # connects; tmux session intact

Layer 3 — Downstream references to the (changed) Tailscale IP

Hostname-first principle: use MagicDNS hostnames (my-mini, my-vps) everywhere; never hardcode 100.x.y.z IPs. After a re-auth changes the IP, grep every peer machine for the OLD IP (health-check scripts, ~/.ssh/config HostName lines, known_hosts, monitoring targets) and hostname-ify each hit.

Full Layer 3 protocol — detection greps, per-location fix table, VPS health-check verification — plus three related failure modes (macOS resolver stuck Not Reachable after Tailscale restart → dscacheutil -flushcache + killall -HUP mDNSResponder; RustDesk/VNC relay blocked by MDM → connect via Tailscale direct-IP; mosh-server: command not found → Homebrew PATH in ~/.zshenv), the worked 2026-05-18 example, and edge-case notes: references/extended-diagnostics.md.

Layer 4 — Dropbox EPERM (FDA on mosh-server)

A distinct mosh-server failure, not an IP-change one: mid-session, every shell access to the Mini's configured Dropbox root returns "Operation not permitted" (EPERM) while the same paths work from the user's own SSH terminal, and disabling the client sandbox does not help. Cause: Dropbox File Provider re-adopted a folder (usually after a rename) → provider-managed files require Full Disk Access, and TCC attributes access to the mosh-server responsible process, not tmux. Fix: grant FDA to /opt/homebrew/bin/mosh-server (+ tmux), then start a new mosh-server (tmux kill-server, fully disconnect + reconnect — TCC grants only apply at process launch). Full mechanism, verification, wrong turns, and prevention: references/dropbox-fda-eperm.md.

Anti-Patterns

• Don't reach for tailscale up --reset first. It resets state on the coordination server, including all advertised routes / exit-node approvals. Use it only when nothing else recovers the daemon.
• Don't kill -9 mosh-servers. A regular SIGTERM (kill PID) lets them release UDP bindings cleanly. -9 skips cleanup; the OS will still release the socket but session state files in /tmp linger.
• Don't restart the whole Mac mini to fix this. Layer 1 + Layer 2 cumulatively take ~3 minutes; a restart adds nothing except risk of losing tmux sessions if anything in them wasn't saved to disk.
• Don't assume which tailscale reflects disk reality. Always cross-check with ls -la $(which tailscale). The PATH cache mismatch is the specific gotcha this skill flags.
• Don't remove /Library/LaunchDaemons/homebrew.mxcl.tailscale.plist during GUI-app cleanup. That's the Homebrew daemon's plist — removing it is what you're trying to avoid.
• Don't apply Layer 1 from a remote SSH session over Tailscale. The tailscale down / tailscale up cycle drops your SSH connection. Either work locally on the Mini's console (Screen Sharing over LAN works), or open a backup SSH session via the LAN IP (ssh user@<mini-LAN-IP>) before starting.

Cross-references

Canonical setup docs (read these first if Layer 1 is unfamiliar)

Doc	Why
docs/setup/mac-mini-setup.md	Mac mini setup of record: Homebrew brew install tailscale is the canonical install (§ Tailscale Remote Access); also covers headless power/sleep settings (sudo pmset -a sleep 0 ...) that make the box stay up without a logged-in user
docs/setup/terminal-setup/terminal-setup.tex	Full terminal-stack manual including § Mosh for Resilient Connections (UDP ports 60000–61000, SSH-keepalive config to prevent Tailscale NAT idleness), § Non-interactive Shell PATH (the mosh-server zshenv gotcha — see Related Failure Mode above), and SSH ProxyCommand pitfalls
docs/reference/terminal.md	Quick reference for the terminal stack (iTerm2, zsh, starship, tmux); config locations and which files sync across machines
docs/guides/tmux-config.md	tmux per-host config; mouse mode is what makes scrollback work through mosh (mosh has no scrollback of its own — tmux underneath holds it)
docs/setup/vps-setup.md, docs/setup/claw-setup.md	Sibling host setups; same Homebrew-Tailscale-mosh pattern adapted to Linux VPS
docs/guides/cross-machine-parity.md	Multi-machine baseline; explains why all four hosts (Mini, UOS, VPS, claw) source packages/dotfiles/shared.zsh
docs/guides/troubleshooting.md	Generic infrastructure troubleshooting — Tailscale/mosh failures get a pointer here from this skill

Related skills

Skill / Rule	Relationship
multi-machine.md rule	Defines mini / my-mini SSH aliases used in examples; mandates relative symlinks across Dropbox; mandates hostname check before machine-specific actions
scheduled-job	Same launchd patterns (this skill's sudo brew services start is a manual variant of what scheduled-job automates for cron tasks)
session-health	Different scope — checks current Claude session state, not network/process state
machine-inventory	Adjacent — inventories what's installed; this skill recovers from a known failure

Origin

Extracted 2026-05-18 from a real diagnostic session — full narrative in references/extended-diagnostics.md §Origin.

Origin

This skill was extracted on 2026-05-18 from a real diagnostic session: the Mac mini's Tailscale GUI app auto-updated, daemon disconnected, headless box couldn't run the GUI daemon, switched to Homebrew/launchd, re-auth gave a new Tailscale IP, existing mosh-servers stranded on the old IP. Original conversation: ~/Library/Mobile Documents/com~apple~CloudDocs/Cloud Downloads/mosh-tailscale.md (iCloud — TCC-gated; copy to /tmp/ to read from Claude Code).