| name | agent-ops-playbook |
| description | Diagnose stuck agents, restart failed processes, manage tmux sessions, run health checks, and resolve common operational failures. |
Agent Ops Playbook
Use this playbook whenever automation appears stalled, degraded, or broken.
1) Diagnose Stuck Agents
Fast Triage
ps aux | rg "openclaw|codex|worker|scheduler|telegram-bridge|run-daemon"
tmux -S ~/.tmux/sock list-sessions
Check Progress vs Stall
tmux -S ~/.tmux/sock capture-pane -t <session> -p | tail -40
Stall indicators:
- same log output across multiple checks
- no CPU usage and no new log lines
- repeating recoverable error with no backoff
2) Restart Crashed Processes
Single Process Restart
pkill -f "<pattern>" || true
nohup <command> >> ~/rick-vault/logs/<name>.log 2>&1 &
tmux Session Restart
tmux -S ~/.tmux/sock kill-session -t <session> || true
tmux -S ~/.tmux/sock new -d -s <session> "cd <repo> && <command>; echo EXIT:$?; sleep 999999"
3) tmux Management Standards
- always use stable socket:
~/.tmux/sock
- use named sessions per workload (
inbox, builder, research, distribution)
- keep completion tail (
echo EXIT:$?; sleep 999999) for postmortem visibility
4) Health Check Procedure
Standard Run
bash scripts/health-check.sh -t ~/.config/openclaw/health-targets.conf --verbose
bash skills/self-healing-ops/scripts/watchdog.sh
If Failing
- identify the first failing dependency
- restart only the failing component
- re-run health checks
- emit an OpenClaw system event with root cause and fix status
5) Log Analysis Patterns
Look for:
- auth failures (
401, 403, token expired)
- network failures (DNS, timeout, TLS)
- rate limiting (
429)
- resource exhaustion (OOM, file descriptor limits, disk full)
Useful commands:
tail -n 200 ~/rick-vault/logs/<service>.log
rg -n "ERROR|WARN|429|timeout|ECONN|auth" ~/rick-vault/logs/<service>.log
6) Common Failure Modes
- missing credentials
- cron not firing
- zombie worker with no progress
- model provider quota exhaustion
- stuck approval queue
7) Escalation Criteria
Escalate immediately when:
- customer-facing production is down and cannot be restored quickly
- data integrity is at risk
- security incident is suspected
- repeated crash loops persist after one restart cycle
