self-hosted-runner-preflight

name	self-hosted-runner-preflight
description	Pre-flight checklist for self-hosted runner failures. Use when investigating low disk, missing runner, or stuck Green Gate on the jleechanorg/worldarchitect.ai fleet. Always verifies host-level container state, not just GitHub API.

Self-Hosted Runner Pre-Flight

When to use this skill

• User says "runner is offline", "runner missing", "CI failed low_disk_space", "Green Gate stuck"
• gh api orgs/jleechanorg/actions/runners shows fewer than 22 runners
• A workflow run has been "in_progress" on Green Gate >30 minutes

The 5-questions pre-flight (MANDATORY before any fix)

For each failure class, answer BEFORE proposing a fix:

1. What does GitHub say? gh api orgs/jleechanorg/actions/runners?per_page=100 --jq '.runners[] | {name,status,busy}'
2. What does the host actually have?
   • mac: docker ps --format "{{.Names}}\t{{.Status}}" | grep org-runner-mac-
   • jeff-ubuntu: ssh jeff-ubuntu "DOCKER_HOST=unix:///home/jleechan/.lima/colima/sock/docker.sock docker ps --format '{{.Names}}\t{{.Status}}' | grep org-runner-"
3. What does the disk look like inside the container? docker exec <name> df -Pk / — needs >=1GB free for preflight to pass.
4. Is the runner in restart loop?
   • docker ps -a --format "{{.Names}}\t{{.Status}}" | grep Restarting
   • If yes, check logs: docker logs --tail 30 <name> for 403/registration errors.
5. Is Green Gate stuck? Check the run step "Check smoke gate (gate 8)" — if IN_PROGRESS >30min with no smoke runs on the branch, see Class C re-dispatch.

Failure-class fixes

Class A: low_disk_space (>=1 runner has <1GB free inside container)

• ROOT FIX: edit self-hosted-oss/pre-job-hook.sh to add disk cleanup (fires on EVERY job, not periodic).
• Pattern:

  AVAIL_KB="$(df -Pk "${RUNNER_WORKDIR}" 2>/dev/null | awk 'NR==2 {print $4}')"
  if [[ "$AVAIL_KB" -lt "$DISK_MIN_KB" ]]; then
    rm -rf /root/.cache/pip
    rm -rf "${RUNNER_WORKDIR}/<repo>"
  fi
  

• Set DISK_MIN_KB=5242880 (5GB, 5x the preflight threshold).
• Belt-and-suspenders: add a periodic launchd script (mac-runner-disk-cleanup.sh style).
• Deploy: bash self-hosted-oss/install.sh syncs to ~/.local/share/worldarchitect-runners/pre-job-hook.sh (bind-mounted live — no restart needed).
• VERIFY: docker exec <runner> bash -c 'pgrep Runner.Worker' returns empty (idle), then df -Pk / shows >=5GB.

Class B: missing runner (compose defines N but only N-1 containers exist)

• mac (ARM64): RUNNER_COUNT=<N> bash ~/.local/share/worldarchitect-runners/start-runner.sh start
• jeff-ubuntu (X64): ssh jeff-ubuntu "cd ~/projects/worktree_runner/self-hosted-colima && DOCKER_HOST=unix:///home/jleechan/.lima/colima/sock/docker.sock docker compose up -d runner-<N>"
• Then verify: docker ps | grep runner-<N> and gh api orgs/jleechanorg/actions/runners --jq '.runners[].name' | grep runner-<N>

Class C: Green Gate stuck on gate 8 (smoke)

• Compare SKEPTIC_GATE_TRIGGER SHA pinned in PR comments to gh pr view <N> --json headRefOid.
• If mismatched, re-dispatch in order:

  gh workflow run "MCP Smoke Tests" --ref <branch> -f pr_number=<N>
  gh workflow run "Skeptic Self-Verify" --ref <branch> -f pr_number=<N>
  gh workflow run green-gate.yml --ref <branch> -f pr_number=<N> -f head_sha=<sha>
  

• All 3 require pr_number; green-gate.yml ALSO requires head_sha.

Class D: token returns 403 from inside container

• Confirm token length: docker exec <name> bash -c 'echo ${#ACCESS_TOKEN}' (should be ~40).
• Test directly: docker exec <name> curl -X POST -H "Authorization: token $ACCESS_TOKEN" -H "Accept: application/vnd.github+json" https://api.github.com/orgs/jleechanorg/actions/runners/registration-token
• 201 → token fine, timing issue (config.sh ran before API ready), re-run start-runner.
• 401/403 → token bad/expired, rotate gh auth token and re-deploy.

Anti-patterns

• Trusting GitHub UI as ground truth for runner health.
• Proposing "schedule a cron to clean up" as the primary fix (use per-job hook).
• Re-dispatching Green Gate without first dispatching smoke (gate 8 polls forever).
• Restarting the container while a job is running (always check pgrep Runner.Worker first).
• Editing the runtime mirror directly (~/.local/share/worldarchitect-runners/) — edit self-hosted-oss/, then bash install.sh to sync.