"Ultracode" = the Workflow tool, never /team-claude teammates (user corrected this twice).
The invoking session AND every teammate lane stall SIMULTANEOUSLY on the
per-account session-limit modal (/rate-limit-options), which never
auto-dismisses after the quota window resets. Liveness+progress monitors miss it
(process alive, no events). Supervisors MUST watch STATE.md staleness (~15
min alarm); on recovery, dismiss the modal in the parent session and have the
lead re-task idle lanes via SendMessage.
**The sidekick and its lanes ALWAYS run as named in-process teammates of the
INVOKING session's Agent Team — visible in the user's panel,
SendMessage-addressable, explicit model on every spawn. There is no tmux
mode (external tmux, codex-engine, and -p sidekicks are banned — user
directive 2026-07-18); durability comes from STATE.md + a P1 resumption bead
Assume the session can hit provider quota at any moment. At mission start AND at every checkpoint, persist tasks/specs/goals/wave-plans in agent-agnostic form (no Claude-specific tool references; plain commands + file paths + PR URLs):
-
Cost-route every agent() call explicitly — verify with a grep, not a memory check. Workflow agents inherit the session model — an unrouted 40-agent fan-out on a Fable session runs entirely on Fable. model: 'haiku' for mechanical merge/cluster/ls stages, 'sonnet' for miners/verifiers/doc writers; main loop (Fable) only for final synthesis. Before launching any script, run grep -n "agent(" <script> and confirm every match either has an explicit model: in its opts or is the deliberate final-synthesis call — an eyeballed "looks routed" pass misses calls added late or inside a loop/helper. Real incident (2026-07-07): a token-usage audit on jeff-ubuntu found one session mixing 428 MiniMax-M3 messages with 367 claude-fable-5 messages — an unrouted fan-out had silently run its mechanical mining/verify stages on the parent session's premium model for weeks before anyone noticed, because nothing checked the script for missing model: before launch. If a stage's cost-routing is genuinely uncertain (e.g. a helper function wraps agent() and the model arg is threaded through indirectly), do NOT assume it's fine — print/log the resolved model for the first agent in each stage at runtime and confirm it in the phase output before trusting the rest of the fan-out.
-
Adversarial verify, refute-by-default. Every candidate finding faces ≥3 independent lenses (e.g. evidence / design / severity) prompted to REFUTE; survive only on ≥2/3 non-refuted. Dedup vs a seen set, not vs confirmed.
-
Audit verifier-death false kills — and detect false-empty completions. A verifier that died (rate limit, API error) is NOT a refutation. After any run with agent failures, re-verify findings whose "refutations" were dead agents before accepting the kill. This generalizes to whole-workflow results: a run that returns 0 confirmed findings is a VOID, not a verdict, if its failures[] list shows mass agent death (e.g. every rejected[].why reads DEAD_VERIFIER/verifier died, or the failures array is dominated by Server is temporarily limiting requests). Distinguish the two before reporting: 0 findings + all agents healthy = real "nothing here" result; 0 findings + mass 429/error failures = the mining/verify stage never actually ran — re-run it, don't ship or record the empty result. Hit twice on 2026-07-07: pr-retro-gapfill (15/15 verify agents died, 5 real Collect-stage findings falsely zeroed out) and code-quality (60/60 verify agents died, 20 real Collect-stage findings falsely zeroed out) — both had genuine candidate findings from a healthy Collect phase; only Verify was VOID.
-
Rate-limit resilience — concurrency is aggregate, not per-workflow. Provider 429s ("Server is temporarily limiting requests") kill whole fan-outs, and the trigger is total agents in flight across ALL concurrently-running swarms on the account, not any single workflow's size (confirmed 2026-07-07: two separate workflows, each individually reasonable, hit 429 in the same window once their combined in-flight count reached ~75). Rules: (a) do not launch a new multi-agent fan-out while another large workflow is mid-fan-out — serialize big stages across sibling swarms, not just within one; (b) on mass 429, stop, start a ~20-min background cool-down timer (sleep 1200 run_in_background or ScheduleWakeup), then retry via a sibling-workflow-as-canary: a tiny throwaway 1-agent workflow (not just "the smallest of the affected workflows") used purely as a health probe, cheaper than blind-resuming the real fan-out; only relaunch/resume the real workflow(s) after the canary succeeds; (c) while cooling down, keep any other concurrently-running lane to ≤4 concurrent sub-agents; (d) stagger large fan-out starts — bursts kill, ramps don't. Root-cause analysis of three separate 429 storms in one evening showed every kill coincided with an INSTANT N-wide parallel() fan-out (16-, 60-, and 15-wide simultaneous cold starts), while gradual ramps (pipeline(), which naturally staggers stage-by-stage) and ordinary serial work were never throttled. Inside a large fan-out, chunk the thunk array into batches of ~8 with a few seconds between batches instead of one parallel() call over the whole array; treat ~16 simultaneous cold starts as roughly the survived ceiling, not a safe target to build up to. Never start two different workflows' fan-out stages within the same minute, even if each individually looks small.
-
Resume semantics. Persist scripts to the session scratchpad (cp from the returned Script file path) so they survive crashes. To resume: TaskStop the run, edit only NOT-yet-completed agent() calls (touching a completed call's prompt/opts invalidates its cache), relaunch with {scriptPath, resumeFromRunId}. Resume journals are same-session only — after a crash, re-run fresh from the persisted script. Cross-session / cross-reboot recovery needs more than the script: write a P1 br bead (see "Bead runbook" above) with session id, script paths, live runIds, and output-dir/doc-numbering state, since /tmp and same-session journals both vanish on a host reboot or /tmp cleanup.
-
Disjoint output dirs per lane. Each workflow writes to its own OUTDIR (docs/plans/<retro>/pr-retro/, .../code-quality/, .../innovations/); parallel workflows must never share a mutable file. Each doc-writer agent may write EXACTLY ONE file.
-
Commit + push artifacts as they materialize — two layers. (a) Each doc-writer agent commits its own single file immediately after writing it (git add <path> && git commit), so partial progress survives even if the workflow itself dies mid-run before ever returning. (b) The workflow's final phase includes one explicit Commit agent that sweeps any stragglers and pushes (git status --short for anything the per-doc commits missed, then git push) and reports the resulting HEAD SHA — never rely on "the doc writers probably committed everything" without a final agent that verifies and pushes. Never leave swarm output only in /tmp or the session dir; repo docs (docs/plans/...) or ~/roadmap/ are the durable homes.
-
Pre-build datasets in the main loop. Miners read a static dataset JSON (e.g. prs-last-month.json built once via gh) instead of each making live API calls — deterministic, cheap, rate-limit-proof.
-
Heartbeats + time-box, cadence scaled to what you're waiting for. Don't default every wakeup to ~270s — that's for tight external polling loops (CI runs, deploys) where the cache stays warm and the state changes fast. For a long multi-agent fan-out (tens of agents, many minutes), poll on a 1200-1800s cadence instead; checking every 270s on a 10-minute workflow just burns cache misses for no new information. Obey the 3-hour autonomy time-box (user-scope rule) regardless of cadence.
-
Finish with /advice. Final package gets an /advice pass (3 parallel reviewers: Opus subagent, /research, /secondo) before the closing PR comment; then /learn.
-
Publishability gate — mandatory final stage, after ALL writer lanes, before the PR is called done. Adversarial verification up to this point attacks candidate findings, not the rendered docs that actually ship — a healthy 3-lens verify pipeline can still publish a doc with a leaked credential path, a stale metric, a forbidden recommendation, or a contradicting sibling doc, because no lane ever re-reads the finished markdown as a whole. Run one final single-agent (or small dedicated) gate over the ENTIRE docset that checks: (a) redaction sweep — rg 'file:///Users|/Users/[a-z]+|ghp_|gho_|x-access-token|serviceAccountKey' over every published doc, replace machine paths/token forms with generic placeholders; (b) cross-doc consistency — every numeric claim in a top-level/summary doc is diffed against its per-finding source doc, disagreements block; (c) freshness re-baseline — every present-tense claim re-checked against the PR's current head SHA, historical evidence explicitly marked "at base <sha>"; (d) supersession markers — each finding has exactly one authoritative doc, earlier superseded drafts carry a banner pointing to it; (e) policy lens — recommendations checked against repo-specific hard law (e.g. ZFC / ZFC-leveling file boundaries, credential-discipline, regex-approval bans) that the evidence/severity/design lenses never covered; (f) recipe validity — every copyable command/acceptance-check states the CORRECT expected outcome (e.g. a negative test should assert red, not green); (g) mechanical hygiene — git diff --check clean. Real incident: a cold review of the 2026-07-06 design-retro docset (~180 agents, 5 workflows, 7/10+ findings confirmed) found all six of these classes present in already-"confirmed" docs — see docs/design-retro-2026-06-adversarial-gaps.md for the full writeup and the six follow-up beads. The gate is planning-scope (a workflow stage + checklist), not production code.
-
Cross-model cold review — MANDATORY, not "when available," for any code-behavior or merge-readiness claim. A same-model swarm shares one set of blind spots: ~180 Claude agents (2026-07-06) unanimously passed a docset in which a codex cold review then found 6 real defect classes, because every swarm agent inherited the same priors about what "verified" means. This recurred on 2026-07-10: a same-model /er + /advice pass (both Claude) signed off on a "fixed" production bug; a single codex adversarial pass immediately found 5 real blocking defects the same-model pass missed entirely — data corruption in an unrelated field, an incomplete guard, meaningless timestamp semantics, unproven causation, and a broken existing test contract — because codex executed the code and reproduced the bugs rather than reading the diff and pattern-matching against the PR's own framing. Before any "ready to merge" / "bug is fixed" / "evidence PASS" claim is reported (not just after a publishability gate on a docset), dispatch at least one DIFFERENT model family as a genuinely adversarial reviewer — codex CLI (codex exec --dangerously-bypass-approvals-and-sandbox), gemini CLI, cursor-agent, whatever is installed (/advice and /secondo already provide multi-model routes for lighter-weight decisions, but a merge-readiness claim on real code needs the full adversarial pass, not the 3-reviewer /advice summary alone). Prompt it explicitly to (a) try to reproduce each defect class by executing code, not just reading it, (b) check for scope creep beyond the PR's stated claim, (c) check whether the evidence proves causation or only correlation. Treat its findings like any candidate: independently reproduce the most concrete/checkable one yourself before accepting the rest at face value (a single spot-check that confirms is strong evidence for trusting the whole report; a spot-check that fails means re-verify everything). Never skip the pass because "the swarm already reviewed it" or "we're in a hurry" — model diversity catches what intra-model redundancy structurally cannot, and skipping it is exactly how a broken fix ships.
-
CI-triggering fan-outs have a SEPARATE capacity ceiling from Claude/provider API rate limits — self-hosted runner/host capacity, not just agent concurrency. Rule 4 covers Claude/Anthropic API 429s from agent concurrency; this is a different resource entirely. Any swarm whose subagents call gh pr update-branch, push commits, or otherwise trigger fresh CI runs can overwhelm a fixed-size self-hosted runner fleet even when the agent fan-out itself is small and well-throttled (each "dumb" agent is one cheap API call, but each successful action can trigger 5-30+ downstream CI jobs). Real incident (2026-07-07, worldarchitect.ai): a 68-item gh pr update-branch sweep (one haiku agent per PR, ~1 minute wall-clock, zero Claude-API rate-limit issues) triggered 26+ simultaneous CI jobs against a 16-runner host and crashed the host outright (hard reboot, uptime reset to ~1min) — the fan-out itself was healthy, the DOWNSTREAM system it triggered work on was not sized for the burst. Before launching any swarm whose action multiplies into external CI/build/deploy triggers, check the target system's current capacity/queue depth first, and batch the trigger in chunks (e.g. ~15-20 at a time with pauses) rather than firing all N at once, even if N is small enough to be a trivial Claude-API fan-out on its own.
-
Max-parallelization sweep — a blocked gate blocks only its dependents, never sibling work (2026-07-13). Under-utilization is a failure mode equal to collision. Three sub-rules, each learned from the DK2D r2 session where 5 disjoint evidence-prep items sat idle for ~an hour while everything waited on one critical-path fix: (a) Standing re-scan: at EVERY coordination touchpoint (lane report, steer, heartbeat, user ping) re-scan the backlog (br ready, open beads, triage tables, STATE next-actions) for dispatchable work that is file-disjoint from active writers — independence is checked continuously, not just at spawn time. (b) Gate scope discipline: when a verdict blocks an artifact (e.g. "evidence run blocked until fix + PASS"), enumerate exactly which items depend on the gated artifact; everything else stays dispatchable — never let a gate expand into a planning-horizon freeze. Prep work that the gated artifact will CONSUME (harness hardening, ledgers, design docs, citations) is the highest-value parallel work because it converts gate-lift into instant execution. (c) Utilization metric in STATE: log lanes-active / items-dispatchable at each touchpoint; if dispatchable > active for two consecutive touchpoints, spawn lanes or launch an ultracode Workflow fan-out (bounded one-shot tasks → Workflow with explicit model: routing; steerable/long-lived → teammate lanes). Foreign/concurrent writers narrow the DISJOINT set — they never justify dispatching nothing.
