Operational health snapshot for the jleechanorg self-hosted runner fleet. Use when user says "check runners", "are runners up", "is jeff-ubuntu down", "diagnose runner health", or before/after runner operations. Produces a structured Markdown report at /tmp/runner-health-<ts>.md with multi-method checks (GitHub API + local Docker + Lima VM + jeff-ubuntu ssh + optional hermes-pc cross-check). Works when jeff-ubuntu is unreachable (key insight: busy=true on runners is itself proof of liveness even when ssh fails).
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Operational health snapshot for the jleechanorg self-hosted runner fleet. Use when user says "check runners", "are runners up", "is jeff-ubuntu down", "diagnose runner health", or before/after runner operations. Produces a structured Markdown report at /tmp/runner-health-<ts>.md with multi-method checks (GitHub API + local Docker + Lima VM + jeff-ubuntu ssh + optional hermes-pc cross-check). Works when jeff-ubuntu is unreachable (key insight: busy=true on runners is itself proof of liveness even when ssh fails).
["GitHub /actions/runners org endpoint shows status (online/offline) and busy (true/false) only — no last_active field. busy=true IS the live-heartbeat proof: a runner can only be busy if it actively polled GitHub and accepted a job.","Fleet is **ezgha-managed** (since 2026-07-06, replacing legacy org-runner): 16 Linux (`ez-runner-b-1..16` on jeff-ubuntu) + 6 mac (`ez-mac-runner-b-1..6` on local Mac) = 22 total expected. Ephemeral by design — each runner runs 1 job then exits, supervisor respawns. Online count fluctuates 19-22 normally; 22/22 steady state is impossible.","When user is on a different wifi than jeff-ubuntu, ssh/ping from local host to jeff-ubuntu fails (different L3 subnet) but the runners themselves are still up if busy=true. Do NOT misdiagnose this as 'host down' — it's 'host unreachable from this network'.","hermes-pc (Slack user U0BC138QXUJ / #hermes-pc channel) is on a separate network (172.20.5.90/19). Use as cross-check when local signals are ambiguous.","Self-hosted-runner-preflight skill already exists for FIX-ORIENTED checks. This skill is different: it's an OPERATIONAL health snapshot (no fix proposed). Use this BEFORE preflight.","MacBook: `ezgha serve` runs as launchd daemon `org.jleechanorg.ezgha` (plist at `~/Library/LaunchAgents/org.jleechanorg.ezgha.plist`, binary at `~/.cargo/bin/ezgha`). Restart: `launchctl kickstart -k gui/$(id -u)/org.jleechanorg.ezgha`.","jeff-ubuntu: `ezgha serve` runs as `systemd --user` unit `ezgha.service` (config at `~/.config/ezgha/config.toml`, count=16). Restart: `systemctl --user restart ezgha`."]
/runner-health — Operational Health Snapshot
When to use this skill
User asks "are the runners up?", "is jeff-ubuntu down?", "show me runner health", "fleet health", "diagnose runners"
Before any operation that depends on the fleet being healthy (e.g., before a /green drive)
After a long gap (wifi change, host reboot) to verify reconnect
Anytime a user reports "all alerts are noise" or "transient alerts" — this skill gives the ground truth
What this skill does NOT do
This is NOT a fix tool. It does not propose remediations.
This is NOT a deep dive. For specific failures, use self-hosted-runner-preflight after this skill reports AMBER/RED.
This does NOT modify any state. It only reads.
The 6 deterministic health checks
Each is a standalone bash script in scripts/ that outputs structured JSON to stdout.
#
Script
What it checks
Failure mode (graceful)
1
check_api.sh
gh api orgs/jleechanorg/actions/runners — 22-runner snapshot + busy count + in-flight jobs (per-repo) + rate limit
exits non-zero on API error; runner fleet unreadable
2
check_docker.sh
local docker ps for mac-side runners + AO auxiliaries
exits non-zero if Docker daemon is down
3
check_lima.sh
limactl list — colima VM status, SSH port, resources
For every GitHub-offline ezgha runner (mac or Linux fleet), cross-checks the local/SSH container state to distinguish session conflict (container Up, GitHub offline — stale registration lock; neither ezgha serve nor ezgha-watchdog detects this, since both only compare local managed-container counts) from ordinary (container also down — ezgha will respawn it). Ported from closed PR #8033's / (bead rev-ws17d).
runner_offline
check_github_session_state()
container_status_for()
exits non-zero on gh api failure; JSON still emitted with error set
6
cross_check_hermes.sh
Posts thread to #hermes-pc and polls for reply (configurable wait)
exits non-zero if Slack MCP unavailable
Session-conflict triage (check 5)
A runner container can show Up X minutes in docker ps (or over SSH into
jeff-ubuntu) while GitHub's API reports status:"offline". This divergence
is a triage signal, not a confirmed diagnosis — check_session_conflict.sh
proves the two states disagree (GitHub-offline + container-running), but it
never reads the container's Runner.Listener log, so it cannot itself
confirm the container is stuck in Runner connect error: Error: Conflict.
Treat every session_conflicts[] entry as "worth a human looking at the
listener log," not as "definitely a stale session." Neither ezgha serve's
own churn-replacement nor the ezgha-watchdog fleet-size check catches this
divergence class — both only compare a local managed-container count
against the configured target, and a session-conflicted container is still
alive and still counted as "managed" locally.
HUMAN-ONLY — do not automate. The remediation below is destructive (it
deletes the runner's GitHub registration and restarts its container,
interrupting any in-flight job on that runner). check_session_conflict.sh
itself stays strictly read-only — do not wire these commands into any
watchdog, cron, or auto-heal path. A human should confirm the
Runner.Listener log actually shows the conflict (or otherwise judge the
runner is safe to bounce) before running any of this by hand:
# Identify the conflicted runner (check_session_conflict.sh JSON: session_conflicts[])
RUNNER_ID=$(gh api orgs/jleechanorg/actions/runners --jq '.runners[] | select(.name=="<name>") | .id')
gh api -X DELETE orgs/jleechanorg/actions/runners/$RUNNER_ID# Mac fleet (ez-mac-runner-*): docker restart <name># Linux fleet (ez-runner-*/ez-canary-runner-*): ssh jeff-ubuntu "DOCKER_HOST=unix:///home/$USER/.lima/colima/sock/docker.sock docker restart <name>"sleep 15
gh api orgs/jleechanorg/actions/runners --jq '.runners[] | select(.name=="<name>") | {name, status, busy}'
A RED|SESSION CONFLICT: ... verdict from runner-health.sh takes priority
over the generic per-arch online-count verdict — see
parse_fields.py::compute_verdict. Runner naming note: ezgha rotates a
generation letter suffix on supervisor restart (observed live: b → c),
so check_session_conflict.sh matches on the stable ez-mac-runner- /
ez- prefixes only, never a hardcoded generation letter.
How to invoke
Direct script (cron / ad-hoc)
# Basic (no cross-check)
bash .claude/skills/runner-health/scripts/runner-health.sh
# With hermes-pc cross-check, 1 call
bash .../runner-health.sh --cross-check 1
# Cross-check max 3 calls (deep investigation)
bash .../runner-health.sh --cross-check 3
Output: console table + Markdown file at /tmp/runner-health-<ts>.md.
As a slash command (via /skill)
User runs /runner-health (or natural-language equivalent like "check runners"). The LLM:
Phase 1 — load context (optional but recommended):
Invoke memory-search skill with query "self-hosted runner jeff-ubuntu health check transient"
Invoke history-search skill to find any prior runner-health invocations in the last 30 days
Note any prior transient-pattern memory entries that should bias the verdict
Phase 2 — run all 5 local checks in parallel (subagents or sequential bash):
Phase 3 — synthesize verdict (GREEN/AMBER/RED) based on the JSON:
Session conflict (highest priority): if check_session_conflict.sh
reports any session_conflicts[], report RED — SESSION CONFLICT with
the affected runner name(s) and the manual-heal steps above instead of
the generic per-arch online-count framing below — ezgha serve /
ezgha-watchdog cannot fix this class.
Fleet shape (ezgha-managed, since 2026-07-06):
16 Linux runners (ez-runner-b-1..16) on jeff-ubuntu — managed by ezgha serve via systemd --user
6 mac runners (ez-mac-runner-b-1..6) on local MacBook — managed by ezgha serve via launchd
Total expected = 22 (16 Linux + 6 mac), but ephemeral by design (each runner exits after 1 job, then re-spawns)
Total online count fluctuates 19-22 normally; a 22/22 steady state is impossible by design
Verdict thresholds (per-arch, since ezgha):
Verdict
Linux online
Mac online
Trigger
GREEN
≥14 (healthy floor)
≥5 (healthy floor)
Both hosts near full, no Docker restarts, all online runners busy=true
AMBER
10–13 (supervisor stuck)
4 (supervisor stuck)
One or both ezgha serve supervisors not replenishing — needs systemctl --user restart ezgha on Linux OR launchctl kickstart -k on Mac
AMBER
any
any
Docker restart loops detected
RED
<10
any
ezgha serve on jeff-ubuntu critically stuck — restart required
RED
any
<4
ezgha serve on Mac critically stuck — restart required
RED
<10 online + 0 busy
n/a
jeff-ubuntu host dark (three-signal: 0 Linux busy + linux_online <10 + SSH unreachable) — DO NOT restart containers, wait for host
CRITICAL preflight rules:
If jeff-ubuntu SSH is unreachable AND Linux busy=0 AND Linux online is also critically low (<10) → it's the host dark class, not a container-level problem. Do NOT restart containers, do NOT spawn new ones. Wait for the host to come back. Log to rev-runn001 for tracking. A high linux_online (e.g. 16/16) with 0 busy + unreachable SSH is a healthy-but-idle fleet on a different wifi subnet, NOT host-dark — it falls through to the GREEN/AMBER verdict below.
If online < expected but ezgha serve is alive (PID exists) and supervisor is just slow → AMBER, not RED. Restart supervisor to force-refill.
Always cross-check by inspecting ezgha status output on the suspect host BEFORE classifying as supervisor-stuck vs host-dark.
Why per-arch instead of total: A single verdict line that says
"RED — 14/22 online" hides whether it's the Mac fleet or the Linux fleet
that's short. Naming the failing arch in the reason makes the fix
obvious — launchctl kickstart on Mac vs systemctl --user restart
on Linux.
Phase 4 — cross-check (only if user asks or AMBER detected):
bash .../cross_check_hermes.sh <0|1|2|3>
Posts to #hermes-pc thread ts (use existing thread or new one), polls for reply, parses response.
Phase 5 — write Markdown report:
bash .../runner-health.sh --write-report (or include in the master script by default)
File: /tmp/runner-health-<unix-ts>.md
Contents: 4-tables (one per check), verdict, recommendations
Phase 6 — present to user:
Print the verdict + table summary to console
Reference the Markdown file path
Output format
Console (default)
=== runner-health @ 2026-06-29 07:50 PDT ===
GitHub API: 22/22 runners online, 11/22 busy (8 Linux, 3 mac)
Docker: 6/6 mac Up, 0 stuck
Lima: colima running, 4GiB mem, 100GiB disk
jeff-ubuntu: unreachable from this host (different wifi subnet)
hermes-pc: cross-checked — 14/22 busy confirmed independently
VERDICT: GREEN — runners healthy, host just on different subnet
Host-dark example (two-signal confirmed, see preflight Class E):
=== runner-health @ 2026-07-02 09:10 PDT ===
GitHub API: 6/22 runners online (all 6 mac), 16/16 jeff-ubuntu Linux runners online=0/busy=0
Docker: 6/6 mac Up, 0 stuck
Lima: colima running, 4GiB mem, 100GiB disk
jeff-ubuntu: ssh timeout + ping timeout — both signals confirm host dark
VERDICT: RED — jeff-ubuntu host dark (16/16 Linux runners offline + SSH/ping timeout confirmed).
NOT a generic <22/22 — this is a single-host outage. See rev-runn001 for tracking; do not open a new bead.
Markdown file (/tmp/runner-health-<ts>.md)
# Runner Health Report — 2026-06-29 07:50 PDT## Verdict: GREEN## Check 1: GitHub API
| Metric | Value |
|---|---|
| Total runners | 22 |
| Online | 22 |
| Busy | 14 |
| Linux busy | 11 |
| mac busy | 5 |
| Rate limit (core) | 4284/5000 |
## Check 2: Docker (local)
| Container | Status |
|---|---|
| org-runner-mac-1..6 | Up |
| AO auxiliaries | 7/8 Restarting (1) (PR #702 fix pending) |
## Check 3: Lima VM
| Field | Value |
|---|---|
| Name | colima |
| Status | Stopped |
| CPUS | 4 |
| Memory | 4GiB |
| Disk | 100GiB |
## Check 4: jeff-ubuntu
| Field | Value |
|---|---|
| Reachable | No (different wifi) |
| Subnet | 192.168.x.x (prior) |
| Recommendation | Check router DHCP lease for new IP |
## Check 5: Session conflict (GitHub offline vs container Up)
| Field | Value |
|---|---|
| Offline count | 0 |
| Session conflicts | none |
## Check 6: hermes-pc cross-check
(only if invoked)
## Notes- PR #702 (AO spawn fix) not yet merged — bake-in bug recurs every ~1h
- 4 PRs open as DRAFT: #702, #8039, #8040, #8041
Conventions
Scripts are independent: each can run standalone for debugging
All scripts output JSON (jq-friendly) and exit non-zero on hard error
Master script runner-health.sh runs all + writes report
Cross-check is opt-in via --cross-check=<0|1|2|3>
Markdown report is in /tmp/ (clears on reboot; user can persist by copying elsewhere)
Anti-patterns
Treating GitHub "online" as proof of liveness — it's registration cache, not heartbeat. Use busy=true.
Diagnosing "host down" from ssh timeout alone — could be routing. Use busy=true as ground truth.
self-hosted-oss/ is retired (removed by PR #8057 and its cleanup follow-up) — do not reference or resurrect it. Runner-fleet changes now go through the ez-gh-actions ("ezgha") daemon / self-hosted-colima/scripts/; see ezgha-watchdog and self-hosted-runner-preflight for that territory.
Storing state in ~/.local/share/runner-health/ — this skill is stateless by design.
