ao-spawn-safety

name	ao-spawn-safety
description	Use BEFORE bulk/iterative AO spawns (>3). Hard cap 20 active workers; spawn ≤5 at a time; never use loadavg as the gate.

AO spawn safety — resource check before bulk spawning

Scope: AO worker spawning only. Do NOT apply these checks to Hermes's own incoming message handling — that blocks Hermes from responding when AO workers are running.

Before issuing any bulk or iterative AO spawn sequence (more than 3 spawns in a loop or in response to an open-ended instruction like "do all" / "take it all the way"):

1. Hard cap at 20 AO workers: If ao list shows ≥20 active sessions total, decline to spawn and report the count instead. This is the single spawn limit — there is no lower "soft pause" threshold. The cap is on AO worker count, not system load average — do NOT use sysctl vm.loadavg as the spawn gate.
2. Batch, don't storm: Spawn at most 5 workers at a time; wait for completions before the next batch.

Why: 2026-05-15 incident — "take it all the way" spawned 517 sessions, loadavg=205, DNS starved, gateway down 2h. The 20-worker hard cap is the guard. (Earlier revisions also carried a soft "pause and ask at 8" rung tied to a kanban.max_spawn config that was never actually set in config.yaml or agent-orchestrator.yaml — removed 2026-06-12 as redundant friction; 20 is the only cap.)

backfillAllPRs — separate per-project cap

backfillAllPRs: true spawns via backfillUncoveredPRs each health-cron cycle (every 3600s by default). Its cap is project.spawnQueue.maxActiveSessions — default 20 (independent of the interactive 20-worker cap above).

Always set this companion when enabling backfill:

backfillAllPRs: true
spawnQueue:
  enabled: true
  maxActiveSessions: 5  # prevents zombie accumulation from dead-tmux sessions

maxConcurrentSessions is NOT a valid config field and is silently ignored by Zod. Using it instead of spawnQueue.maxActiveSessions has no effect.

Why 9 zombies accumulated (2026-06-18): Dead tmux sessions → hasSession()=false → PR marked "uncovered" → new worker spawned each cycle. Without a per-project cap (or with a silently-ignored field), the default 20-cap was never hit and accumulation continued unchecked.