| name | host-diag |
| description | Diagnose tailnet host health and reachability for the infra fleet - grab a post-incident host-diag snapshot, auto-capture one on each dead->alive SSH recovery via host-watch, probe node-global pressure (conntrack/load/apiserver), and read fleet reachability across the k3s pair and towers. Use when a host flaps, an SSH hop goes dark, or a reachability row goes red. Aliases - host diagnostics, host-diag, host-watch, fleet reachability, node pressure, tailnet host down, SSH flap, conntrack. |
host-diag
Status: 🛠 Runbook | Last updated: 2026-07-03
Judgment layer over the tailnet host-health tools. All four are read-mostly (the
scripts run sudo reads - ss, journalctl, dmesg, conntrack -L - and write only
to /tmp). Pick by symptom. These scripts had no docs before this skill - it is their
home now.
Grab a host-diag snapshot
When a host had an incident (SSH went dark, k3s or tailscaled restarted, an OOM, a
conntrack/netfilter drop) and you want one frozen picture of its state. Stream it in -
it never gets copied to the box:
ssh <host> -- bash -s < scripts/host-diag.sh
It tees to stdout and /tmp/host-diag-<ts>.txt on the remote. Sections: META,
load/memory/PSI, listening sockets, sshd state, recent unit restarts, ssh + k3s +
tailscaled journals (last 2h), kernel ring, OOM hits, netfilter/conntrack, iptables,
fail2ban. Skim scripts/host-diag.sh for the exact section list.
Auto-capture on recovery (host-watch)
When a host is flapping and you cannot sit on it - host-watch polls its Tailscale SSH,
logs every state transition, and on each dead -> alive edge streams host-diag into
the remote so you catch the state right as it came back:
ward exec host-watch host=<alias>
Tune with POLL_INTERVAL (default 15s) and OUT_DIR (default
/tmp/host-watch-<alias>). Recovery snapshots land as recovery-<ts>.txt beside
watch.log. Contract in scripts/host-watch.sh. Probes by MagicDNS short name,
key-only.
Probe node pressure
When you suspect node-global exhaustion behind a control-plane or SSH flap (timeouts
not refusals -> dropped SYNs -> conntrack, CPU/softirq, or NAT pressure, not a
pod-local fault). Run it ON the node under load:
scripts/node-pressure-probe.sh <label> <seconds> [interval] [outdir]
It samples conntrack%, load-per-cpu, and apiserver /livez latency to a CSV and prints
a peak summary. Method + the 2026-06-12 measurement are in the §7 traps runbook
docs/k3s-deploy-notes-traps-runner.md.
Read fleet reachability
The reachability rows are the truth for "is the mesh up". Two rollouts, two docs (kept
in docs/ because the fleet-reachability Ansible roles own them):
docs/fleet-reachability.md - the ser8 + kai-server k3s pair. A red row localizes
which node lost the mesh.docs/fleet-reachability-towers.md - host-level WSL/Windows towers. A Windows-green /
WSL-red split localizes a WSL-bridge fault, not a real outage.
