orchestration

name

orchestration

description

Run delegated work reliably — writing subagent charters, verifying subagent claims, keeping judgment at the orchestrator tier, ledger discipline, recovering from interruptions, and shipping/merging lane PRs. Use when spawning subagents or workflows, coordinating parallel lanes, resuming after a session limit or compaction, or rebasing and merging a lane's PR. The subagents skill covers task sizing and model choice; this skill covers everything after the spawn.

Orchestration

Delegation only works when the orchestrator treats every subagent claim as unverified and every judgment call as its own. These rules were each paid for by a real incident.

Charters

A charter is the task prompt a subagent receives. Requirements:

• Reference every document by absolute path — a relative path resolves differently (or not at all) in the agent's working directory.
• When the work depends on an external spec or vendor document, fetch it raw — curl or a real browser, never a summarizing fetch — into a scratchpad cache file, and name that cache path in every dependent charter. All lanes then build from identical bytes; left to fetch it themselves, each agent works from its own paraphrase and the lanes drift apart on the contract they were supposed to share.
• When parallel lanes could touch the same module, name the sibling lanes and give each an explicit do-not-touch list of files/functions the others own. For genuinely shared append-only files (routes config, nav layout, seeds), say up front that rebase conflicts there resolve keep-both. Before launching, check each item's designed fix against its lane's do-not-touch list: an item whose only defensible fix lives in a sibling-owned file moves to the lane that owns the file, or gets an explicit carve-out — left as is, the builder must choose between skipping the item and violating the boundary, and one wave produced a skipped item, a banked residual, and a flagged two-file deviation from exactly this.
• When parallel lanes allocate from a shared identifier pool — seed namespace prefixes, fixture ids, ports, queue names — the orchestrator publishes one canon allocation table in every lane's charter before the first launch, and each lane sub-partitions its allotment across its own flows. Lanes that pick identifiers independently collide, and the collisions surface as another lane's spec failing long after both lanes went green in isolation; one wave shipped three separate collisions (two lanes sharing a phone prefix, two sharing a fixture id, one lane colliding with itself) before the table became standing practice.
• When several lanes will each touch a shared computed constant — a contract fingerprint, a generated schema hash — every charter declares the same resolution protocol up front: on rebase, recompute the value by the mechanism that owns it (its own test's computation), never keep-mine or keep-theirs. Three parallel lanes once rewrote the same fingerprint off stale bases, and every merge needed a hand repin. Each charter also declares whether its own items are expected to move the constant; an unexpected movement is a stop condition to report, never a silent repin — a lane that repins a public contract hash it had no business moving ships the change invisibly under a green suite.
• Assign each subagent an explicit scratch-file namespace — its lane or task name as a filename prefix — and forbid it from writing any shared scratchpad file, above all the orchestrator's session ledger, whose actual path the charter must give since it is named for the effort rather than by a fixed convention. Parallel agents that share a scratch filename clobber each other's state, and a fix agent that writes its own notes over the session ledger destroys the orchestrator's recovery record.
• Include a stop-on-contradiction clause: if the charter's factual premise (a claimed bug, a claimed missing feature, a spec citation) doesn't survive verification against the primary source, the agent halts that item and reports with citations instead of inventing. "Verified already correct, no change made" is a valid, expected outcome. The clause covers the charter's designed fix shape and its measured file:line anchors, not only the finding: sibling merges stale both between charter-writing and execution, so a builder that re-verifies the design against its own worktree and deviates with citations is complying — the deviation is adjudicated at merge, never treated as disobedience.
• A charter touching product-monolith instructs the agent to read product-monolith/docs/process/engineering-conventions.md (absolute path in the lane's worktree) before coding and to treat it as binding, naming the repo's designated reference app as the reference implementation; a reviewer or skeptic charter for product-monolith names that file as the codified-house-rule source for rubric (b) below. A charter that writes or reshapes tests also names docs/process/testing.md as binding and does not prescribe a test mechanic itself — a charter once offered a patch-based interleave that file forbids, and only the builder's own diligence avoided shipping it.
• A charter that includes a mutation proof (deliberately breaking code to show a test goes red) also specifies the restore protocol: restore byte-identically, verify via git status/git diff, and purge __pycache__ under the touched packages — a same-size restore within the same mtime second leaves stale bytecode that quietly poisons every later run in that tree.
• When two reference sources disagree, the charter names the conflict for adjudication — it never silently picks one. A charter's factual claims are no more trustworthy than a builder's until verified.
• Builders never run long gates. A charter ends at commit (or commit + push) and STOP, with only scoped foreground checks before it (the touched module's tests, lint) — "run the full suite in the foreground" does not survive contact with builders: they background it and end their turn terminally stalled, because a workflow agent's turn ending is final. The orchestrator runs full gates itself as its own background shell (its re-invocation on completion costs nothing) and owns push/PR/merge when the charter stops at commit.
• Write each new charter/script fresh with the Write tool. Deriving one by programmatically splicing a previous one introduces silent syntax errors.
• Relay a finding at exactly the scope it was measured. "These two named tests survive the mutation" is not "the suite survives the mutation"; widening it on the way into a charter sends the agent to disprove a claim nobody made, and its correction can bury the narrower gap that was real. Quote the finding's own boundary, and mark anything you extrapolated as yours to verify.
• Cross-check a charter's exit assertions against its own items before launching: an exit gate demanding a file show no diff contradicts another item that edits the same file, leaving the agent to choose between violating the gate and dropping the item. Scope byte-identity checks to the exact lines or symbols they protect, never to whole files other items touch.
• A parameterized script must parse args defensively (string-or-object) and hard-throw on any missing required field — a silently-undefined interpolation produces a plausible-looking empty result that reads as a clean pass.
• A slice that ships a route, loader, or form must drive that surface live at least once — a minimal browser interaction or a real-Request smoke — before handoff. A unit-only exit bar passes code whose query crashes at runtime or whose form cannot submit (a select with no name attribute).
• A skeptic or verifier charter carries TWO acceptance rubrics, not one: a finding stands if either (a) its concrete-input failure scenario occurs at runtime, or (b) it identifies a violation of a codified house rule or doctrine in the work under review. A runtime-only rubric refutes correct doctrine findings — an ad hoc query fragment inlined in a view where product-monolith/docs/process/engineering-conventions.md prescribes a model-owned queryset method — exactly when the fix is cheapest. Doctrine findings against unmerged schema or shape default to fix-now, because schema shapes are effectively permanent once merged.

Verify, never trust

• Re-run the gates yourself before advancing any stage on a subagent's self-report.
• A completeness claim — "every route covered", "every call site migrated", "every consumer updated" — is only checkable against a denominator derived live from the system of record (the URL resolver, the schema, a repo-wide census), never against the plan that produced the work: the plan omits exactly what the work omitted. Establish the denominator before declaring any "every X" objective met, and prefer a gate that recomputes it and fails on drift over a registered list or document describing coverage — registered state duplicates the system and goes stale silently.
• Reconcile numbers exactly: baseline + computed delta = measured, and identical across two runs. A mismatch is a hard stop, not noise. A collect-only reconciliation carries one extra term: registry-parametrized suites expand on their own whenever the registry they parametrize over grows, so account for that expansion before reading a collect-count jump as drift.
• Reconcile a builder's charter item by item against its diff, not only by counts: every charter item ends in exactly one visible state — implemented in the diff, verified-already-correct, or an explicitly reported skip. Builders silently drop items, and numeric reconciliation cannot catch an item whose omission changes no counts (a test reshape, a mechanical conversion) — one such drop shipped unnoticed and only surfaced in a later personal read of the merged diff.
• Reconcile the adjudicated rulings against the union of the lane charters before the wave launches: every ruled fix appears in exactly one lane's items. Orchestrators drop whole rulings the same way builders drop charter items — three ruled fixes once reached no lane at all, caught only by a ruling-by-ruling reconciliation during the final write-up.
• A charter's expected gate numbers are measured on the current base tip at charter-writing time, never carried forward from per-lane history — every sibling merge stales them. When lanes have merged since the last measurement, re-measure the union before chartering anything that asserts a count.
• A task counts as launched only when its task/run id from the tool result is in the ledger. A written script or a narrated intent is not a running task.
• Never block on a running subagent or workflow — a synchronous wait for its output freezes the conversation until the agent finishes, so the user can neither interject nor redirect, and the user has had to interrupt one to say so. Launch in the background, keep the chat free for whatever else is pending, and act when the completion notification arrives. When the harness's task registration is uncertain (after a compaction), arm a background watcher on the task's own artifacts rather than polling in the foreground.
• Independent lanes launch simultaneously, never in staggered waves. Host contention (CPU, database) slows every lane's gates but never justifies holding a launch back — an idle lane costs more than a slow gate. Contention is managed inside the running lanes instead: full test suites queue under the concurrency cap the skill sets.

Judgment stays at the top

• Design and adjudication are orchestrator-tier work. When the evidence under a design changes (stale reference, corrected spec), redo the design at the orchestrator tier — never hand the stale design down with a "re-verify your citations" instruction.
• When a lane empirically disproves a factual premise in a shared research digest, correct the digest itself immediately and visibly, before the next lane reads it. Sibling lanes read the artifact, not the report that refuted it, so every later lane inherits the same wrong premise from a document that still asserts it.
• Grounding priority when state or policy is uncertain: primary sources (git, live files, the actual spec) beat live skills, and live skills beat ledger notes or compacted memory. On a policy detail, the live skill file always wins over ledger shorthand.
• When parallel lanes independently invent shapes for the same shared surface, adjudicate one canon and give later-shipping lanes an explicit adopt-the-canon duty at rebase. Distribute the canon as a verbatim, byte-identical file package every lane applies unchanged — identical bytes auto-merge everywhere, so the shared file never conflicts again.
• When lanes re-express existing behavior (authorization gates, validation rules), builders guess wrong in both directions — stricter and looser both shipped from the same wave. Commission one audited map of every capability to its exact source-of-truth behavior (with file:line citations) before any fix pass, and make every fix charter cite the map, never intuition.
• Bank out-of-scope findings (with repro evidence) into a running list for a dedicated pass — never fix them inline, never drop them. Before deferring an item until some future event, check that the event does not itself depend on the item: work parked until a readiness decision it is a precondition for never restarts, because the decision waits on the work and the work waits on the decision.
• A designer's "too invasive / out of scope" blast-radius claim is a factual claim, not a judgment call — grep it before accepting the compromise it justifies.
• Whether to fix a known defect or a regression is not the user's decision. Once verification has established that something is broken, fixing it is the only defensible answer, so asking spends the user's attention on a question with one outcome and stalls the work until they happen to be present. Bring the user a genuine choice between shippable alternatives, or a trade-off only they can price — never a request for permission to remove a defect.
• Before asking the user to rule on a surface or planning a change to it, check the repo for in-flight work: gh pr list plus a scan of remote branches. An open PR may have already settled the question — a dependency queued for a user decision had already been removed by a PR in flight, and only the user's correction caught the stale premise.
• A question's premises about the live system — which domain serves what, what sits behind a proxy, what a live response carries — are probed live before the question reaches the user: a browser, , DNS. Production configuration overrides every default in the repo, so a code-derived claim about production is a guess, and "not determinable from the repo" is a prompt to probe, not a finding — one question shipped with both domains' roles wrong from a code-default inference, and the user had to say "open them in a browser and see for yourself".

Browser-agent hygiene

• Every agent that drives a browser ends with its browser session closed, and the orchestrator verifies it — a successful close, or an agent's report that it closed its session, is not evidence that the processes died; only a process listing is. After each browser agent completes, and again between waves, sweep to zero with the shipped command:

  bash .claude/scripts/browser-sweep.sh          # list survivors with their ages; exits 1 if any are alive
  bash .claude/scripts/browser-sweep.sh --kill   # kill each survivor by its exact pid, then re-list to prove it
  

• Judge leakage by process count and age, never by reported RSS — paged-out memory hides an order of magnitude (7.6GB visible while roughly 80GB was actually held).

• Leakage comes from sessions accumulating across sequential waves, not from parallel width. Keep lanes parallel; the invariant is verified teardown, not a concurrency cap.

• Kills are the sweep's job precisely because it kills by exact pid after listing what is about to die. Never reach for a pattern kill (pkill -f and relatives) as cleanup or as verification — a pattern that reads as lane-scoped routinely matches unrelated long-running processes on this machine, and listing the pids is the inspection step a pattern kill skips.

Ledger discipline

• Structure the ledger as a STANDING DIRECTIVES head (settled policy, kept current) plus a chronological log. Reground after compaction from the head AND the tail — a tail-only reground lets settled directives fade. Reground with bounded reads — the head block, then targeted slices — never by reading the whole file: a mature ledger runs hundreds of kilobytes, and one full read consumed enough context to force an immediate re-compaction.
• Record launches with their task/run ids, lanes with their base commits, and verdicts with their evidence.
• When a long effort concludes, externalize the durable record (decisions, divergences, audit results) to a permanent artifact — a GitHub issue or PR body — before the scratchpad is cleaned up.

Recovery after an interruption

Step 0 is always to positively enumerate what is alive (task list, workflow roster) — a missing state file or empty output file usually means still-running, not dead. The harness task list can come back empty after a compaction or a host sleep while the underlying shells still run: ps for the actual processes and check whether their log files are still growing before declaring anything dead. Declaring a live task dead and relaunching into its worktree corrupts both. Then, for tasks with positive evidence of death:

• Audit the worktree first: git status/git log. Clean tree at a known commit → relaunch the charter unedited. Dirty tree → launch a continuation agent on the same worktree whose first step is diffing working tree vs last commit to classify done/partial/untouched, then finishing — never a blind restart.
• A QA agent killed mid-mutation-proof can leave deliberately broken code on disk — check for a live mutation before anything else runs there.
• An agent that is still addressable resumes by message from its last checkpoint, keeping its context, instead of restarting cold.
• Probe returned capacity by attempting real work. Never idle-wait.

Shipping and merging a lane

• A PR opens only after the lane's full pre-PR pipeline has run: builder claims verified, full gates green, the adversarial code-review audit complete, and an orchestrator ruling on every confirmed finding. Small diffs are not exempt — momentum after green gates is exactly when the audit step gets dropped, and on a three-line security fix the almost-skipped audit was the one that surfaced a fourth same-class vulnerability.
• Write every PR title and description for an external reader: someone with zero knowledge of the execution that produced the change must understand what it is and why it matters. No orchestration vocabulary (goals, lanes, charters, sanctioned allowances, finder/verifier counts, workspace skills or tooling-repo names), and no superseded project state — describe what the change IS in the product's own terms, and rewrite the body as the work evolves so it always reads as current truth, not history.
• An umbrella PR's body is a deliverable, not a tracker. While work is landing it may be organized around progress, but the effort is not done — and a goal is not met — until the body is rewritten to read as finished: no tracking framing ("Landed", "updated as areas land", in-flight lists), and no links to the internal PRs that built the branch — sub-PR references are noise to an external reader, who needs the result described, not the delivery.
• When a PR whose title promises no behavior change (test, docs, chore types) actually changes production behavior, the title must say so and the body must lead with a complete list of every behavior change — readers won't expect behavior in that diff. Enumerate the list from the real base-to-tip diff of non-test paths, never from memory of the work.
• A defect fix described only by its new behavior reads as housekeeping. Each behavior-change entry states the problem first — what broke, under what conditions, with what consequence — and then the fix, so its real weight is legible: "visit ingestion is idempotent" hides that a routine SQS redelivery permanently broke a paying customer's ad attribution. Write the consequence in the product's terms, not the schema's.
• A before/after contrast in a PR body ("no longer", "used to", "restored", "remains") is only tellable against the PR's base: if the base itself carries the old behavior, the contrast describes the change; if the branch introduced the surface, the contrast narrates the branch's own development history, which the reader never saw — state current truth only. Check each such phrase against the base, not against memory of how the work went. A refute pass does not catch this class on its own — fact-checkers verify claims as stated and confirm a contrast that is factually true but tells internal history — so make base-tellability its own explicit rubric item in any refute pass over a body.
• A branch that backs an open PR — a lane under review, a shared feature branch — syncs by merging its base in, never by rebase or force-push. Rebase is for branches nothing else references yet.