ai-agent-multi-agent-coordination

name	ai-agent-multi-agent-coordination
description	Use when designing systems where multiple agents collaborate on one task — supervisor/worker, debate, plan-execute, peer handoff. Covers handoff message contracts, shared scratchpad, conflict resolution, deadlock detection, and the cost/SLA discipline that keeps multi-agent setups from spiralling.
metadata	{"portable":true,"compatible_with":["Codex","codex"]}

AI Multi-Agent Coordination

Acknowledgement: Shared by Peter Bamuhigire, techguypeter.com, [phone].

Use When

• A single agent cannot reliably do the job (too many tools, too many domains, ambiguous routing).
• Standing up a supervisor + workers pattern (one planner, several specialists).
• Implementing a debate pattern where two agents propose answers and a judge picks (or a critic refines).
• Designing plan-execute (one plan-only agent, one execute-only agent) for separation of concerns.
• Implementing agent-to-agent handoff (sales agent hands to billing agent mid-task).

Do Not Use When

• A single agent can do the task — use it. Multi-agent multiplies cost and latency and divides reliability. Default to single-agent.
• The task is just multi-tool — that's not multi-agent; one agent with N tools is still one agent.
• The task is a deterministic workflow with LLM steps — ai-agent-runtime-architecture §1 decision.

Required Inputs

• Single-agent design first. Only escalate to multi-agent when single-agent demonstrably fails on eval.
• Agent runtime (ai-agent-runtime-architecture).
• Tool registry (ai-agent-tool-catalogue-and-action-gating).
• Cost / step budgets (ai-agent-cost-and-step-budgets) — multi-agent budgets are an order of magnitude larger.

Workflow

1. Read this SKILL.md.
2. Justify multi-agent (§1). Reject it as default.
3. Choose a coordination pattern (§2): supervisor/worker, debate, plan-execute, peer-handoff.
4. Design the handoff message contract (§3). See references/handoff-protocols.md.
5. Implement the shared scratchpad with versioning (§4).
6. Implement conflict resolution for divergent proposals (§5). See references/conflict-resolution.md.
7. Implement deadlock detection (§6).
8. Apply cost discipline (§7) — multi-agent budgets are per-aggregate, with kill-switches.
9. Apply anti-patterns (§8).

Quality Standards

• Multi-agent justified by a written rationale + eval data comparing to single-agent.
• A named supervisor (or named protocol) coordinates. No leaderless free-for-all.
• Handoffs are structured messages with schemas, not natural-language guesses.
• Shared state is versioned; concurrent writers use optimistic concurrency.
• Conflict resolution rule is deterministic, not "another LLM call". Use voting, scoring, or human escalation.
• Total step budget covers the aggregate, not per-agent. One agent can starve others if budgets are local.
• Deadlock detection runs every N seconds; circular waits abort with a structured failure.
• Multi-agent runs are traced as one task with sub-spans per agent. Replay must reconstruct the whole.
• Cost-per-task is measured and reported; if > 3× single-agent baseline, the multi-agent design is reviewed.

Anti-Patterns

• "Crew of agents" with no supervisor and no protocol — emergent loops, infinite back-and-forth.
• Handoff via free-text "@billing_agent please help" parsed by regex. Fragile, non-replayable.
• Shared scratchpad as a global mutable dict in memory — lost on worker crash, no audit.
• "Critic" agent that just re-runs the planner with "be better". No measurable improvement, doubled cost.
• Per-agent budgets without an aggregate cap. One agent burns its budget; supervisor spawns another; infinite spawn.
• Multi-agent for a task the eval shows a single agent handles with > 90% success. You bought 3× cost for no quality gain.
• Two agents writing to the same resource without idempotency / conflict detection. Race conditions become bugs that look like hallucinations.

Outputs

• Multi-agent justification document (single-agent baseline + eval comparison).
• Coordination pattern decision.
• Handoff message schema per agent pair.
• Shared scratchpad schema + concurrency model.
• Conflict resolution rule.
• Deadlock detector spec.
• Aggregate cost / step budget policy.

Evidence Produced

Category	Artifact	Format	Example
Architecture	Multi-agent topology spec	Markdown + diagram	docs/ai/multi-agent-topology.md
Correctness	Handoff protocol tests	CI report	tests/ai/handoff/
Release evidence	Multi-agent vs single-agent eval	Markdown	docs/ai/eval/multi-vs-single.md
Operability	Deadlock incident runbook	Markdown	docs/runbooks/multi-agent-deadlock.md

References

• references/supervisor-worker.md — supervisor/worker contract.
• references/handoff-protocols.md — agent-to-agent handoff message schemas.
• references/conflict-resolution.md — conflict and deadlock resolution.
• Companion: ai-agents-tools, ai-agent-runtime-architecture, ai-agent-tool-catalogue-and-action-gating, ai-agent-cost-and-step-budgets, ai-agent-observability-and-replay, ai-agent-eval, distributed-systems-patterns.

§1 Justify Multi-Agent

Multi-agent is 5-50× the cost of single-agent, 2-10× the latency, and substantially harder to evaluate and debug. Justify before adopting.

Acceptable justifications:

• Domain partition is sharp — sales agent and billing agent need different prompts, tools, models, and personas.
• One agent cannot fit all required tools (> 15 tools degrades selection).
• Adversarial review (debate) measurably improves quality on the eval suite vs single-agent + critic-prompt.
• Different agents need different model tiers (planner = flagship, worker = distilled, judge = mid).

Unacceptable:

• "Crews are cool right now."
• "We want it to feel like a team."
• "Maybe we'll need it later."

The justification doc lives in docs/ai/multi-agent-justification-<feature>.md and includes the eval data.

§2 Coordination Patterns

Pattern	Topology	When
Supervisor / Worker	One planner; N specialist workers; supervisor synthesises	Multiple domains, clear partition
Plan-Execute	One planner agent emits a structured plan; one executor agent runs it	Separation of concerns; planner can be a stronger model
Debate	Two agents propose; a judge picks (or a synthesiser merges)	Quality-critical tasks; eval shows debate improves baseline
Peer Handoff	Agents pass control via structured messages; no central supervisor	Conversation-style flows (sales → billing → support)

See references/supervisor-worker.md for the dominant pattern.

§3 Handoff Message Contract

Handoffs are not free-text:

{
  "handoff_id": "h_2026_0511_001",
  "from_agent": "sales_agent",
  "to_agent": "billing_agent",
  "task_id": 12345,
  "tenant_id": 42,
  "reason": "Customer requested an invoice for last month's hours.",
  "context_summary": "ACME, May 2026, 32 hours at $400/h, billing contact [e-mail].",
  "structured_state": {
    "customer_id": 88,
    "billing_period": {"start": "2026-05-01", "end": "2026-05-31"},
    "computed_hours": 32,
    "rate_usd_per_hour": 400
  }
   
   
   

The receiving agent reads structured_state as ground truth and context_summary as advisory. The receiving agent does not trust the free-text fields to make irreversible decisions; it re-validates via its own tools.

Full protocol in references/handoff-protocols.md.

§4 Shared Scratchpad

A versioned key-value store per task:

CREATE TABLE agent_scratchpad (
  task_id      BIGINT NOT NULL,
  key          VARCHAR(128) NOT NULL,
  value        JSON NOT NULL,
  version      INT NOT NULL DEFAULT 1,
  written_by   VARCHAR(64) NOT NULL,
  written_at   DATETIME NOT NULL,
  PRIMARY KEY (task_id, key)
);

Writes are optimistic-concurrency: UPDATE ... WHERE version = :expected_version. Mismatch → caller refetches and merges.

Agents read snapshots; writes go through a runtime helper that records the writing agent's identity.

§5 Conflict Resolution

Two agents propose different answers. The deterministic resolver:

def resolve(proposals):
    # Group by canonical answer
    groups = group_by(proposals, key=canonicalise)
    if len(groups) == 1:
        return groups[0].representative
    # Voting if odd number of judges
    if len(proposals) >= 3 and len(proposals) % 2 == 1:
        majority = max(groups, key=lambda g: len(g.members))
        if len(majority.members) > len(proposals) / 2:
            return majority.representative
    # Tie-breaker: confidence score
    return max(proposals, key=lambda p: p.confidence)

For high-stakes tasks (irreversible actions, money), conflict → human escalation, not another LLM call.

§6 Deadlock Detection

Build a wait-for graph from handoff history:

def detect_deadlock(task_id):
    handoffs = db.query("SELECT from_agent, to_agent FROM handoffs WHERE task_id = ?", task_id)
    g = build_directed_graph(handoffs)
    cycles = find_cycles(g)
    if cycles:
        abort_task(task_id, reason=f"deadlock_cycle: {cycles[0]}")
        page_oncall(...)

A scheduled job runs every 30 seconds. Default deadlock detection threshold: any cycle that has had ≥ 3 traversals in the last 5 minutes.

§7 Cost Discipline

Aggregate budget: task.cost_budget_usd and task.step_budget cap the sum across all agents in the task, not per-agent. A child agent's costs decrement the parent's budget.

When budget hits 80%, the supervisor's next call is annotated BUDGET_PRESSURE_80 so the supervisor can summarise and exit early.

When budget hits 100%, the task transitions BUDGET_EXCEEDED regardless of which agent is mid-step.

§8 Anti-Patterns

• Multi-agent as default. It's an escalation, not a starting point.
• Free-text handoff. Loses replayability.
• Per-agent budgets without an aggregate cap. Spawn loops.
• Critic agent that doesn't measurably improve outcomes.
• No deadlock detection. Tasks hang for hours.
• Shared scratchpad in process memory. Lost on crash.
• Agents writing to the same external resource without coordination. Race conditions become "hallucinations".
• "Agents will figure it out." They will not.