runner-health

name	runner-health
description	Operational health snapshot for the jleechanorg self-hosted runner fleet. Use when user says "check runners", "are runners up", "is jeff-ubuntu down", "diagnose runner health", or before/after runner operations. Produces a structured Markdown report at /tmp/runner-health-<ts>.md with multi-method checks (GitHub API + local Docker + Lima VM + jeff-ubuntu ssh + optional hermes-pc cross-check). Works when jeff-ubuntu is unreachable (key insight: busy=true on runners is itself proof of liveness even when ssh fails).
type	skill
scope	repo
owner	$USER
version	1.0.0
triggers	["check runners","are runners up","is jeff-ubuntu down","diagnose runner health","runner status snapshot","fleet health","verify ubuntu runners"]
allowed-tools	["Bash","Read","mcp__slack__*","Skill (memory-search, history-search)"]
context	["GitHub /actions/runners org endpoint shows status (online/offline) and busy (true/false) only — no last_active field. busy=true IS the live-heartbeat proof: a runner can only be busy if it actively polled GitHub and accepted a job.","Fleet is **ezgha-managed** (since 2026-07-06, replacing legacy org-runner): 16 Linux (`ez-runner-b-1..16` on jeff-ubuntu) + 6 mac (`ez-mac-runner-b-1..6` on local Mac) = 22 total expected. Ephemeral by design — each runner runs 1 job then exits, supervisor respawns. Online count fluctuates 19-22 normally; 22/22 steady state is impossible.","When user is on a different wifi than jeff-ubuntu, ssh/ping from local host to jeff-ubuntu fails (different L3 subnet) but the runners themselves are still up if busy=true. Do NOT misdiagnose this as 'host down' — it's 'host unreachable from this network'.","hermes-pc (Slack user U0BC138QXUJ / #hermes-pc channel) is on a separate network (172.20.5.90/19). Use as cross-check when local signals are ambiguous.","Self-hosted-runner-preflight skill already exists for FIX-ORIENTED checks. This skill is different: it's an OPERATIONAL health snapshot (no fix proposed). Use this BEFORE preflight.","MacBook: `ezgha serve` runs as launchd daemon `org.jleechanorg.ezgha` (plist at `~/Library/LaunchAgents/org.jleechanorg.ezgha.plist`, binary at `~/.cargo/bin/ezgha`). Restart: `launchctl kickstart -k gui/$(id -u)/org.jleechanorg.ezgha`.","jeff-ubuntu: `ezgha serve` runs as `systemd --user` unit `ezgha.service` (config at `~/.config/ezgha/config.toml`, count=16). Restart: `systemctl --user restart ezgha`."]

/runner-health — Operational Health Snapshot

When to use this skill

• User asks "are the runners up?", "is jeff-ubuntu down?", "show me runner health", "fleet health", "diagnose runners"
• Before any operation that depends on the fleet being healthy (e.g., before a /green drive)
• After a long gap (wifi change, host reboot) to verify reconnect
• Anytime a user reports "all alerts are noise" or "transient alerts" — this skill gives the ground truth

What this skill does NOT do

• This is NOT a fix tool. It does not propose remediations.
• This is NOT a deep dive. For specific failures, use self-hosted-runner-preflight after this skill reports AMBER/RED.
• This does NOT modify any state. It only reads.

The 6 deterministic health checks

Each is a standalone bash script in scripts/ that outputs structured JSON to stdout.

#	Script	What it checks	Failure mode (graceful)
1	check_api.sh	gh api orgs/jleechanorg/actions/runners — 22-runner snapshot + busy count + in-flight jobs (per-repo) + rate limit	exits non-zero on API error; runner fleet unreadable
2	check_docker.sh	local docker ps for mac-side runners + AO auxiliaries	exits non-zero if Docker daemon is down
3	check_lima.sh	limactl list — colima VM status, SSH port, resources	exits non-zero if limactl missing
4	check_jeff_ubuntu.sh	`ssh -o ConnectTimeout=5 jeff-ubuntu 'uptime; df -h /home; free -h; DOCKER_HOST=... docker ps	wc -l'`
5	check_session_conflict.sh	For every GitHub-offline ezgha runner (mac or Linux fleet), cross-checks the local/SSH container state to distinguish session conflict (container Up, GitHub offline — stale registration lock; neither ezgha serve nor ezgha-watchdog detects this, since both only compare local managed-container counts) from ordinary (container also down — ezgha will respawn it). Ported from closed PR #8033's / (bead rev-ws17d).