| name | ainb-fleet:daemon |
| description | Long-running watcher that scans every claude session every 5s and
auto-sends `continue` to any session whose recent tmux pane buffer
matches a known API-error regex (rate_limited, overloaded_error,
internal_server_error, request_timeout, socket_hang_up, fetch_failed,
ECONNRESET). Use this when you want unattended recovery from
transient API failures across the fleet.
|
| version | 0.1.0 |
| user-invocable | true |
| triggers | ["ainb-fleet:daemon","fleet daemon","auto-continue","fleet watcher","recover stuck sessions"] |
| allowed-tools | ["Bash"] |
ainb fleet:daemon
Superseded by /ainb-fleet:atc for managed fleets. ATC
absorbs this daemon's job — its ERR playbook does the same auto-continue, but
with a per-session retry cap + escalate-on-exhaustion that this daemon
lacks (see the sharp edge below). Prefer ainb fleet atc setup <name> for
unattended supervision. The daemon stays for unmanaged one-off recovery; do
not run it against sessions an ATC instance already manages — they race.
Background watcher. Reads each session's tmux pane (capture-pane) and
auto-continues sessions hitting API errors via tmux send-keys. It does
not register as a peer — it is purely tmux-driven (broker health is
checked only to print an informational line).
Run
ainb fleet daemon
ainb fleet daemon --verbose
For real background use:
nohup ainb fleet daemon --verbose > ~/.ainb-fleet.log 2>&1 &
What it does each tick (every 5s)
- Re-discover all sessions (ainb + peers + jobs, merged + deduped).
- For each tmux-bearing session:
tmux capture-pane -p -S -80 (last 80 lines).
- Run the API-error regex set over the buffer.
- If a match fires AND the (session_id, pattern, snippet-tail) dedupe
key hasn't been seen yet → send
continue to that session via the
standard route (tmux-first by default — see AINB_FLEET_TRANSPORT).
Detected error patterns
| name | regex |
|---|
rate_limited | \brate[_ ]limited\b |
overloaded | \boverloaded_error\b | \bModel is overloaded\b |
internal_server_error | \binternal_server_error\b |
request_timeout | \brequest timed out\b |
socket_hang_up | \bsocket hang up\b |
fetch_failed | \bAPI Error\b | \bfetch failed\b |
connection_reset | \bECONNRESET\b | \bconnection reset\b |
Case-insensitive. Word-boundaried to avoid false positives.
Dedupe key
key = (session_id, pattern_name, last_40_chars_of_match_context)
Within a single daemon run, the same key only fires continue once.
This prevents spam when the error text scrolls up but is still in the
buffer.
⚠️ Sharp edge — no retry cap in v0.1
If a session is permanently broken (wrong credentials, model deprecated,
loop bug), the daemon will keep firing continue forever as new errors
match new dedupe keys. Watch the daemon log; kill it (Ctrl-C or
kill <pid>) when you see runaway repetition.
Roadmap: per-session retry cap with exponential backoff.
Stop the daemon
pkill -INT -f "ainb fleet daemon"
kill <pid>
The daemon holds no broker registration, so there is no stale peer entry to
clean up on exit — Ctrl-C / kill is enough.
Observe what the daemon is doing
On start the daemon prints whether the broker is reachable (informational
only — writes go via tmux regardless):
[fleet/daemon] broker healthy at 127.0.0.1:7899
# or, when the broker is down:
[fleet/daemon] broker not reachable — operating in tmux-only mode
With --verbose, every detection also prints:
[fleet/daemon] auto-continue -> <tmux_session> (<pattern>)
Tail the nohup log to watch in real time:
tail -F ~/.ainb-fleet.log
When NOT to run the daemon
- During hand-debugging — the daemon will send
continue while you're
reading the error, racing your investigation.
- On a fleet doing batch work that explicitly throws errors as control
flow — the daemon will misinterpret intentional errors.
- On a session with no live tmux pane (bg job, dead tmux) — there is
nothing to send-keys to; under the default
tmux-first it can only
reach such a session if it still has a healthy broker peer.
