This is a HARD GATE. Do not produce ANY output, code, strategy, design, or recommendation without completing this research.
Before you act, you MUST execute every applicable research step. Research-before-acting is the difference between professional work and amateur guessing:
#
Research Step
Why It Matters
Where to Look
RP1
Verify domain currency. Check for breaking changes, deprecations, new standards, or version shifts since the knowledge cutoff.
[STALE_RISK] Outdated advice breaks real systems. API deprecations, framework version bumps, and security advisory changes happen continuously. Outputting based on stale knowledge damages credibility and produces broken results.
Official docs, changelogs, GitHub releases, RFC tracker
RP2
Audit the system or codebase. Read relevant files. Understand existing patterns, constraints, and architecture before proposing changes.
[CONTEXT_VIOLATION] Solutions that ignore existing patterns create technical debt. A change that contradicts the established architecture is worse than no change — it introduces inconsistency that compounds over time.
Cross-reference claims against authoritative sources. Every factual assertion needs a verifiable source. Mark each: [VERIFIED], [COMPUTED], or [ESTIMATED].
[HALLUCINATION_GUARD] Claims without sources are indistinguishable from hallucinations. The #1 cause of incorrect output is treating assumptions as facts. Source tagging prevents this.
Official documentation, peer-reviewed papers, RFCs, specifications
RP4
Identify known failure modes. Before recommending, list what commonly breaks. For each failure mode: trigger condition, detection signal, and mitigation.
[FAILURE_BLINDNESS] Every domain has known failure patterns. Output that doesn't address them is dangerously incomplete. If you cannot name 3+ failure modes for your recommendation, you don't understand it well enough to recommend it.
Quantify impact in concrete units. Replace abstract claims ("faster," "better," "more scalable") with exact numbers, even if estimated.
[VAGUENESS_PENALTY] "Faster" is unverifiable. "Reduces p95 latency from 340ms to 120ms (±15ms)" is verifiable. Abstract adjectives hide ignorance behind confidence. Concrete numbers expose gaps.
Benchmarks, production metrics, pricing data, published performance data
RP6
Map side effects and downstream impacts. What else breaks? Which dependencies are affected? Which downstream consumers need updating?
[CASCADE_BLINDNESS] Changes to one component ripple outward. A fix in module A can break module B that depends on A's old behavior. Map the blast radius before acting.
Dependency graph, cross-skill coordination table, API consumers list
RP7
Verify against non-negotiable quality gates. What are the minimum quality bars for this domain (accessibility, security, performance, accuracy, compliance)?
[QUALITY_FLOOR] Every domain has minimum standards below which output is invalid regardless of functionality. Missing WCAG AA = broken. Leaking credentials = broken. Silent data loss = broken.
Declare explicit limitations and edge cases. What does this NOT handle? What are the known boundaries? What scenarios are explicitly out of scope?
[SCOPE_HONESTY] Declaring limitations is a feature, not an admission of weakness. It prevents misuse, sets correct expectations, and demonstrates true understanding. Every solution has boundaries — naming them is professional.
This SKILL.md, domain literature, edge case databases
If you skip any of these research steps, you are not producing quality output — you are guessing with confidence. Guessing wastes time, breaks systems, and destroys trust. The references, ground rules, and decision trees in this skill exist specifically to prevent guessing. Use them.
Compliance: Research must be executed before any substantial output. For each step, document findings inline in your response using [RESEARCHED] marker: [RESEARCHED: RP1 — Domain verified against changelog v2.4. No breaking changes since cutoff.]. Partial research = partial quality. Zero research = zero credibility.
🔄 Iterative Research Loop — Research at EVERY Decision Point, Not Just Entry
The RP1-RP8 cycle above is NOT a one-time gate. It fires continuously at every material decision point throughout the workflow:
Loop
When It Fires
What Re-research Validates
Loop 0: Pre-Action
Before producing ANY output, code, strategy, or recommendation
At every adjustment, phase transition, scale-out, or significant state change
Has the context changed? Are the original assumptions still valid? Has new information invalidated the Loop 0 conclusions?
Loop 2: Pre-Exit
Before closing, handing off, escalating, or declaring completion
Is the deliverable complete by the quality gates defined in RP7? Are all limitations declared (RP8)? Have failure modes been addressed (RP4)?
Loop 3: Post-Action
After completion: compare expected vs. actual outcome
What was the efficiency ratio (actual / theoretical max)? What learnings emerged? What should be fed back into the pattern database for future decisions?
Integration into Core Workflow:
Every decision point in a skill's Core Workflow must be marked with:
[RESEARCH LOOP: Re-execute RP1-RP8 before proceeding to next phase]
This ensures the agent pauses to re-verify ALL research dimensions before making the next decision. A skill that only researches at entry and then operates on auto-pilot is a skill that makes decisions on stale context.
Markers for output: At each loop, the agent outputs: [RESEARCHED: Loop N — RP1-RP8 re-verified. Key delta from previous loop: ...]
Why this matters: A decision made in Loop 0 may be catastrophically wrong by Loop 2 because the context changed. Markets move. Requirements shift. Dependencies update. The research loop catches context drift before it becomes output error.
Compliance: Research must be executed before any substantial output AND re-executed at every decision point. For each research loop, document findings inline. Partial research = partial quality. Zero research = zero credibility. Stale research = dangerous confidence.
Route the Request
Route multi-agent design through the topology patterns in Section 3 and decision trees in Section 11. If < 3 agents, use Section 11 for simple delegation. If ≥ 3 agents, use full topology + typed state from Sections 3-4.
Ground Rules — Read Before Anything Else
Never let agents share mutable state without a typed schema. Never delegate without explicit success/failure contracts. Never run agents without observability instrumentation. Never exceed 3 delegation hops — use flat topologies. State corruption in multi-agent systems costs $100K+ per incident.
Anti-Hallucination
Admit uncertainty: If you are unsure about any API, version, configuration, or domain-specific fact, state "I am not certain about X — consult [authoritative source]" rather than guessing.
Flag your knowledge cutoff: State "My training data ends in [date]. Verify current documentation for any version-specific details or newly released features."
Never guess security: If you are uncertain about cryptographic defaults, auth configurations, or compliance thresholds, refuse to guess and point to the official security documentation.
VERIFIED: Mark all definitive claims with [VERIFIED] when confirmed by documentation. Mark uncertain claims with [BEST-KNOWN] and provide the citation path to verify.
Operational Ground Rules
#
Negative Constraint
Mechanical Trigger (detect before executing)
Violation Response
R1
ANCHOR to runtime versions before generating framework-specific code. Never generate Fastify/Express/Django/FastAPI/Prisma/SQLAlchemy API calls from training data alone — your training data may be stale.
Trigger: skill receives code-generation task involving framework-specific APIs → run scripts/runtime-version-detect.sh [project-root] --skill-context to detect installed versions → if detection succeeds, anchor all API calls to detected versions → if detection fails, request version info from user
STOP. Respond: "Detected: {runtime}@{version}, {frameworks}@{versions}. Anchoring all API calls to these versions. I will add // VERIFY: comments on any API call where the detected version is newer than my training cutoff."
R2
RUN the ROI Gate before any non-emergency code change. Every code change that is not (a) a security fix, (b) a compliance requirement, or (c) an active production incident must pass scripts/roi-gate.sh. If the gate returns negative, refuse to write the code.
Trigger: skill receives a code-generation or refactoring task that is NOT a security fix, compliance requirement, or production incident → estimate implementation cost in engineer-hours → compare against annual value of the change → if cost > value, gate fails
STOP. Respond: "ROI Gate analysis: This change costs approximately $[X] to implement but saves $[Y]/year. Payback period: [N] years. If payback > 2 years, I recommend declining this work. See scripts/roi-gate.sh for the full formula."
The Expert's Mindset
You design agent systems assuming every handoff will fail. You enforce typed contracts, idempotent delegation, and cost-aware topology selection. You treat agent output as stochastic — never deterministic.
Operating at Different Levels
2-3 agents: Simple supervisor or sequential topology
5-10 agents: Hierarchical with typed state and LangGraph
10-50+ agents: Swarm with CrewAI/AutoGen, cost-optimized routing
Cross-team: Federation with agent-handoff-protocol handoff contracts
When to Use
Use when designing multi-agent systems with 3+ collaborating agents, debugging agent state corruption, optimizing multi-agent costs, or scaling from prototype to production agent swarms.
Decision Trees
Decision Tree 1: Agent Topology Selection
┌── INPUT: Number of agents and task structure
│
┌────┴────────────────────┐
│ │
▼ ▼
Sequential pipeline? Parallel independent tasks?
│ │
▼ ▼
HIERARCHICAL SWARM / PEER-TO-PEER
(Supervisor + Workers) (Self-organizing)
│ │
├─ Use when: clear ├─ Use when: tasks are
│ dependency chain │ independent, agents
├─ 3+ agents │ are homogeneous
├─ LangGraph: StateGraph ├─ Odd number of peers
│ with conditional edges │ (3, 5, 7) — avoid 2
└─ Risk: supervisor └─ Risk: deadlock on
bottleneck consensus failure
Decision Tree 2: State Management Strategy
┌── INPUT: State complexity and sharing needs
│
┌────┴────────────────────┐
│ │
▼ ▼
Simple key-value state Complex typed state
(few fields, one writer) (many fields, multi-writer)
│ │
▼ ▼
Message-bus dict TypedDict + Pydantic
(AutoGen default) (LangGraph default)
│ │
│ ┌────┴────┐
│ │ │
│ ▼ ▼
│ Single owner Shared mutable
│ per field? state needed?
│ │ │
│ ▼ ▼
│ Field ownership Immutable snapshots
│ + write guard + merge-after-complete
Decision Tree 3: Delegation Mode Selection
┌── INPUT: Agent interaction pattern
│
┌────┴────────────────────┐
│ │
▼ ▼
One-way handoff Bidirectional dialogue
(fire-and-forget) (negotiation, Q&A)
│ │
▼ ▼
Direct delegation Conversation loop
│ │
├─ Use when: task is ├─ Use when: agents must
│ self-contained │ reach consensus
├─ Set max_depth = 3 ├─ Set max_turns = 20
├─ Reject cyclic ├─ Implement convergence
│ delegation chains │ detector
└─ Structured output └─ Tiebreaker: supervisor
required or human escalation
Decision Tree 4: Cost Optimization Strategy
┌── INPUT: Agent token consumption pattern
│
┌────┴────────────────────┐
│ │
▼ ▼
Cost per task > $5? Agent utilization < 50%?
│ │
▼ ▼
AUDIT delegation chains RIGHT-SIZE agent model
│ │
├─ Check for infinite ├─ Simple tasks → fast model
│ loops (cycle detection)├─ Complex reasoning →
├─ Verify context │ high-capability model
│ pass-through protocol ├─ Idle agents → spin down
└─ Reduce max_turns └─ Track tokens-per-task
as primary KPI
If a command or approach fails, follow this escalation path before giving up:
Symptom
First Action
If That Fails
Last Resort
Tool/command not found
Check installation: which [tool] or [tool] --version. Install via package manager (brew install, npm install -g, pip install)
Check PATH: echo $PATH. Verify the tool binary is in a PATH directory. Symlink or update PATH if installed but unreachable
Use a functionally equivalent alternative tool. If rg is unavailable, use grep -r. If gh is unavailable, use git directly or the GitHub API via curl
Permission denied
Check ownership: ls -la [path]. Fix with chmod or sudo if appropriate. For API errors (401/403), verify credentials haven't expired: echo $TOKEN or check ~/.netrc
Refresh credentials: re-authenticate with the service. For file permissions, check if the file is locked by another process: lsof [path]
Request elevated permissions or use a different authentication method (token vs password, SSH key vs HTTPS)
Command hangs or times out
Kill the process: Ctrl+C. Re-run with a timeout: timeout 30 [command] or gtimeout on macOS. Check system resources: top, df -h, netstat -an
Add verbose/debug flags: --verbose, --debug, -v. Check logs: tail -f [logfile]. Reduce scope: process fewer files, query a smaller time range, limit concurrency
Split the work into smaller batches. Implement a retry loop with exponential backoff (1s, 2s, 4s, 8s). If the issue is network-related, add --retry 3 or equivalent
Unexpected output or error message
Read the error message completely — the solution is often in the last 3 lines. Search the exact error: grep -r "[error text]" in the repo to find prior occurrences
Check GitHub issues for the tool: gh issue list --repo owner/repo --search "[error keyword]". Check Stack Overflow
Simplify the approach. If the complex one-liner fails, break it into 3 sequential commands. If the specialized tool fails, use a more basic tool with more steps
Data integrity concern (wrong output, silent failure)
Verify with a manual check: compare output against a known-correct baseline. Add assertions: `[command]
grep -q "[expected]" && echo "OK"
Hard failure boundary: If 3 different approaches all fail, STOP. Do not iterate infinitely. Log what was tried, capture the error output, and report the blocking issue with full context. Move to the next independent task rather than blocking all progress on one failure.
Gotchas
Gotcha
Cost
Fix
Agent loop without a turn limit — two agents enter a negotiation pattern where Agent A asks "Are you done?", Agent B replies "Almost, one more thing," and they repeat 500 times until hitting the token limit. Half the context window is "Almost, one more thing."
$10K-$30K in wasted API costs per incident when a multi-agent conversation consumes $200 in tokens without producing output. Over a month of development iterations, this happens 5-10 times.
Set a hard maximum turn count (default: 20). Implement a convergence detector: if the last 3 turns don't produce new, actionable information, terminate the conversation and escalate. Log turn count and token usage per agent conversation as KPIs.
Agent delegates to another agent without passing sufficient context — Agent A says "Fix the auth bug in the user service" but doesn't pass the stack trace, the failing test case, or the git blame for the last change. Agent B starts from scratch, re-discovers the bug, and arrives at a different fix that reintroduces an older regression.
$15K-$40K per incident in redundant investigation time plus regression risk. The fix is worse than the original because the second agent lacked the context the first one had.
Implement a context pass-through protocol: every delegation message must include (1) the original problem statement, (2) what's already been tried, (3) log/error output, (4) relevant file paths with line numbers, and (5) the hypothesized root cause. Never delegate with less than these five elements.
Parallel agents operate on the same file without coordination — Agents A, B, and C each read config.yaml, each modify it differently, each write it back. Only the last write survives. The changes from A and B are silently lost.
$20K-$50K in lost work and corrupted state when concurrent file modifications are lost without detection. In the worst case, the corruption isn't discovered until a deployment fails hours later.
Implement file-level locking: before an agent writes to a file, it acquires a lock (flock, advisory lock, or explicit coordinator check). If lock is held by another agent, wait or escalate. Prefer sequential phases that operate on non-overlapping files. Run git diff --stat after all agents complete to verify no unexpected collisions.
Agent failure is silent — Agent C encounters a tool error and returns empty output. The orchestrator interprets empty output as "nothing to do" and proceeds. Three hours later, the orchestrator "completes" successfully while Agent C's assigned task (security scanning, data validation) was never performed.
$30K-$100K in undetected failures when a critical agent silently drops out. If the security scan agent fails silently, code ships un-scanned. If the data validation agent fails silently, corrupt data propagates.
Require explicit output from every agent: each agent must return a structured result with status (success/failure/partial) and artifact list. Orchestrator validates that every expected agent produced a result. Empty output = failure, not success. Implement a heartbeat check: if an agent hasn't produced output in 5 minutes, poll its status.
Best Practices
Do select topology based on agent interdependence, not agent count — A 3-agent system with circular dependencies needs a supervisor; a 10-agent system with fully independent tasks runs efficiently in peer-to-peer. Topology mismatch costs 30-50% more tokens from redundant communication and unnecessary resolution loops. Map the dependency graph (who needs whose output) first, then select topology by minimizing the total communication edges. Topology is architecture, not configuration.
Prefer typed shared state over unstructured message-passing for agent coordination — Message-passing without a schema degrades into ad-hoc JSON that agents parse inconsistently. A typed state (Pydantic model, LangGraph TypedDict, Protocol Buffer) is self-documenting, machine-verifiable, and catches field-level ownership conflicts at startup rather than at runtime. A runtime state corruption incident from schema mismatch costs 4-12 engineer-hours to debug and fix — validation at startup costs zero.
Always enforce a maximum delegation depth with cycle detection — Unbounded delegation chains create circular reasoning loops where Agent A delegates to Agent B who delegates back to Agent A. A 3-hop limit with cycle detection (reject delegation to any agent already in the active chain) prevents infinite loops that burn $50-$500 in API tokens before anyone notices. Each additional delegation hop compounds hallucination probability by 15-20% as context fragments.
Never use even-numbered peer groups for consensus decisions — Two peers deadlock on disagreement with no resolution path. Four peers split 2-2 with gridlock. Always use odd-numbered groups (3, 5, 7) with a predefined tiebreaker: supervisor override or majority vote with timeout escalation. A deadlocked 4-agent peer group burns tokens indefinitely with zero progress — at $0.05-$0.15 per agent-turn, that's $12-$36/hour of wasted API spend.
Measure token-per-completed-task as the North Star efficiency metric — Track total tokens consumed (all agents, all delegation hops, all conflict resolution) divided by completed tasks. If token-per-task grows >20% week-over-week without a topology or task-complexity change, you have a silent delegation leak or context bloat. Target: <5,000 tokens per simple single-domain task, <20,000 per complex cross-domain multi-agent task. Alert on 3 consecutive weeks of growth.
Production Checklist
Before deploying or delivering work from this skill, verify:
#
Check
Verify
☐
Topology documented: agent roles, communication edges, data flow direction, and failure mode per edge mapped in architecture artifact
Architecture diagram shows all agents with labeled edges; failure mode for each edge documented (timeout, rejection, corruption)
☐
Typed state schema enforced at startup: every state field has declared owner agent and readers list; Pydantic/LangGraph validation runs before first task
Run schema validation → zero fields claimed as owner by multiple agents; type mismatches caught at schema load, not at runtime
☐
Max delegation depth configured (default: 3) with cycle detection that rejects any agent already present in active delegation chain
Simulate 4-hop delegation attempt → verify rejection at hop 4 with clear error; simulate cycle (A→B→A) → verify rejected with cycle-detected error
☐
Convergence detector active: if last 3 agent turns produce no new information, escalation fires instead of continuing loop
Simulate stuck agent producing repetitive output → verify escalation fires after 3 non-productive turns; escalation includes full context
☐
Cost tracking dashboard operational: per-agent token consumption, latency P50/P95, and success rate measured; alert on anomalies
Dashboard shows per-agent metrics with 5-minute granularity; alert fires if any agent consumes >50% of total budget without measurable progress
☐
Failure modes tested: hallucination cascade, infinite delegation loop, and supervisor bottleneck all produce clean escalation with audit trail, not silent failure
Simulate all 3 failure modes in staging environment → verify each produces structured escalation log with root cause and recommended remediation
☐
File-level concurrency safety: no two agents write to the same file simultaneously; locking or sequential phasing prevents write collisions
git diff --stat after multi-agent production run shows zero unexpected file collisions; file-locking audit log shows no contention timeouts
☐
Rollback plan is documented and tested
Checkpoint-based rollback tested: restore state to checkpoint N-1 after injecting simulated corruption; replay from clean state with zero data loss; recovery time documented
Verification
#
Complete when...
Verify
☐
Complete when Topology documented with agent roles, communication edges, and failure modes
Architecture diagram shows all agents, data flow, and escalation paths
☐
Complete when Typed state schema defined with field ownership per agent
Pydantic/LangGraph state model: each field has owner and readers list
☐
Complete when Max delegation depth set (default: 3) with cycle detection
Delegation chain tracker rejects if target agent is already in chain
☐
Complete when Max turn count enforced (default: 20) with convergence detector
Last 3 turns produce new information; otherwise escalate
☐
Complete when Context pass-through protocol implemented for every delegation
Each delegation includes: problem, tried, logs, file paths, hypothesis
☐
Complete when File-level locking or sequential phases prevent concurrent write collisions
git diff --stat after multi-agent run shows zero unexpected collisions
☐
Complete when Structured output required from every agent (status + artifacts)
Empty output treated as failure; heartbeat check polls idle agents every 5 min
☐
Complete when Cost tracking: tokens-per-task measured and optimized per agent
Dashboard shows cost breakdown; idle agents spun down
When this domain goes wrong, it goes wrong in predictable ways. Here are the most common failure signatures, their root causes, and the fix you'll reach for after you've been burned once.
Symptom
Root Cause
Fix
Lesson
Supervisor agent delegates to sub-agent → sub-agent delegates back to supervisor with "I need more context" → supervisor delegates again → infinite loop burns $500 in API tokens in 8 minutes before timeout kills the pipeline
No cycle detection. Supervisor and sub-agent both have delegation as their default escalation path. When a sub-agent is uncertain, it delegates "up" — which is back to the supervisor, who delegates "down" to another sub-agent who might also delegate back. No max-depth counter
Implement max delegation depth (default: 3). Track delegation chain: agent ID → agent ID → agent ID. Reject delegation if the target agent is already in the current chain. Add a "resolve or escalate to human" terminal state — if max depth reached, the current agent must produce a best-effort answer, not delegate further
Delegation without cycle detection is an infinite money-burning machine. Every agent in the chain must know the full delegation path and refuse to create cycles. The max-depth counter is the circuit breaker — when it trips, the agent must produce output, not delegate again
Peer-to-peer topology: Agent A and Agent B disagree on architecture decision. Both escalate to "decide by consensus" but there's no tiebreaker — only 2 peers. Both post "waiting for consensus" and the pipeline hangs indefinitely
P2P topology requires an odd number of peers or a tiebreaker. With exactly 2 peers, a disagreement creates deadlock. No timeout on consensus — the pipeline just waits. No escalation path from peer-level deadlock to a supervisor or human
Always use odd-numbered peer groups (3, 5). Define a consensus timeout: if no agreement after 3 rounds of debate, escalate to supervisor agent or human-in-the-loop. P2P with 2 agents is not a topology — it's a deadlock architecture. Minimum viable peer group is 3
Peer-to-peer with an even number of peers is a design bug. Every disagreement becomes a deadlock. The topology must define: how are ties broken? What's the timeout? Who escalates? Without these answers, P2P is just a distributed hang
TypedDict state: Agent B modifies results.summary field. Agent A depends on results.summary being in the format it wrote. Agent B's modification changes the data type from List[str] to str. Agent A's next invocation crashes with AttributeError: 'str' object has no attribute 'append'
Shared mutable state with no write protection. TypedDict defines the schema but doesn't prevent an agent from writing a value that breaks another agent's expectations. Agent A assumes it "owns" the summary field; Agent B doesn't know about this assumption because there's no field ownership model
Define field ownership in the state schema: each field has an owner agent and readers list. Readers can read but not write. Implement immutable snapshots: agents receive a frozen copy of state, not a mutable reference. Changes are collected and merged after the agent completes. Validate state schema after every agent invocation — reject state that doesn't match the contract
Shared mutable state without ownership is the root of all multi-agent bugs. TypedDict defines the shape but not the access control. Every field needs: who writes it, who reads it, and what happens when a non-owner writes it (reject with error). Schema validation after every agent turn is not optional
Swarm topology with 8 agents → 3 agents become idle, never assigned work. The swarm self-organizes around the first 5 agents that claim tasks; the other 3 wait for tasks that never arrive. Total cost is 8 agents' worth of tokens for 5 agents' worth of work
Swarm self-organization without load balancing. Agents claim tasks on a first-come basis. Fast-starting agents claim multiple tasks; slow-starting agents get none. No mechanism to redistribute work from overloaded to idle agents. Idle agents still consume resources
Implement work stealing: idle agents poll for unclaimed tasks and also poll for "overloaded agent" signals. Set per-agent task caps: no agent can claim more than N concurrent tasks. Add a swarm coordinator that monitors task distribution and reassigns if imbalance exceeds 2:1. Track agent utilization — idle agents should be spun down, not left polling
Swarm topology without load balancing is just paying for idle compute. Self-organization optimizes for speed, not fairness — the first agents to claim tasks get all the work. Work stealing and task caps are the minimum fairness mechanisms. An idle agent that's not contributing should be decommissioned, not left burning tokens
Hallucination cascade: Agent 1 fabricates an API endpoint (/api/v2/users/bulk-delete). Agent 2 builds a client library with that endpoint. Agent 3 writes integration tests. Agent 4 deploys to staging. The endpoint doesn't exist — it was hallucinated. Entire pipeline produces working code for a non-existent API
No inter-agent fact verification. Each agent trusts the previous agent's output as ground truth. Agent 1's hallucination propagates through the chain because no agent independently verifies external facts. The pipeline optimizes for internal consistency, not external correctness
Implement verification checkpoints: agents that consume external API descriptions must verify them against actual API specs or live endpoints. Cross-validation: when Agent 2 receives an API endpoint from Agent 1, it must call GET /api/v2/openapi.json to verify the endpoint exists. Consensus check: if 2+ agents independently resolve the same fact and disagree, flag for human review
A hallucination in a single-agent system produces wrong output. In a multi-agent system, it produces a cascade of wrong outputs that all agree with each other — internally consistent, externally false. Verification checkpoints between agents are the only defense against hallucination propagation
Conflict resolution with majority vote: 5 agents vote on architecture decision. Result: 2 for option A, 2 for option B, 1 abstains ("insufficient context"). Pipeline hangs waiting for a majority that will never come. Timeout kills the pipeline with no decision and no fallback
Voting requires a majority threshold but the pipeline didn't define what happens when no majority is reached. 5 agents with 3 options is a common deadlock scenario. No tiebreaker, no timeout escalation, no "supervisor override" fallback
Define a decision ladder: (1) Unanimous → decide. (2) 2/3 supermajority → decide. (3) Simple majority with tiebreaker → supervisor agent decides. (4) No majority after timeout → escalate to human. Set a per-decision timeout: 3 debate rounds or 10 minutes. The pipeline must never hang — it must always produce a decision or an escalation
Voting systems need tiebreakers. The failure mode of majority voting with even numbers is not "bad decision" — it's "no decision." Every voting mechanism must define the fallback when the vote fails: supervisor override, human escalation, or default-to-safest-option. A hung pipeline is worse than a suboptimal decision
State Log
This skill maintains a decision ledger to prevent context drift and ensure recall across sessions. Every major architectural choice, constraint decision, and trade-off must be recorded so that subsequent agents (or future sessions) can recover context without replaying the entire conversation.
What Good Looks Like
All agents share typed state via LangGraph/Pydantic. Every delegation has idempotency and timeout. Full OpenTelemetry traces across agent boundaries. Cost per task is measured and optimized. Zero silent state corruption.
Deliberate Practice
Design a 5-agent hierarchical topology for a code review pipeline. Implement typed shared state in LangGraph with Pydantic schemas. Simulate 3 failure modes (hallucination cascade, infinite loop, supervisor bottleneck). Build a cost dashboard tracking tokens-per-task across agents.
References
See the References section at the end of this skill and references/ directory for deep-dive reference files on LangGraph, CrewAI, AutoGen, and swarm patterns.
1. Problem Statement
Multi-agent systems fail silently without deliberate orchestration. When 3+ agents collaborate, you encounter: state corruption across handoffs ($100K+ in inconsistent decisions), hallucination cascades where downstream agents amplify upstream errors ($500K+ wrong architecture), infinite delegation loops ($50K+ wasted compute), and supervisor bottlenecks that cap throughput ($200K+ degraded SLAs).
This skill provides the architecture, protocols, and failure mode prevention to build production multi-agent systems across LangGraph 0.2+, CrewAI 0.30+, AutoGen 0.4+, and OpenAI Swarm.
Portability target: All patterns work with Claude Code, Copilot CLI, Cursor, OpenClaw, Gemini CLI.
def resolve(agents: list, threshold: float = 0.5) -> str:
votes = Counter(a.propose() for a in agents)
total = len(agents)
for option, count in votes.most_common():
if count / total >= threshold:
return option
return escalate_to_human(votes)
8. Observability & Instrumentation
8.1 What to Measure
Metric
Instrument
Threshold
Inter-agent latency
handoff_start → handoff_end
< 500ms P95
Delegation depth
depth counter per chain
≤ 5
State hash drift
compare hashes pre/post handoff
Must match
Hallucination score
cross-agent consistency check
< 0.3 divergence
Token consumption / agent
per-agent token counter
Budget per task
Idle agent time
last_active timestamp
Evict if > 30s idle
8.2 Decision Audit Trail
from langfuse import Langfuse
trace = Langfuse().trace(name=f"multi-agent:{task_id}")
for handoff in handoff_chain:
span = trace.span(
name=f"handoff:{handoff.source}→{handoff.target}",
metadata={
"depth": handoff.depth,
"state_hash": handoff.hash,
"latency_ms": handoff.latency_ms
}
)
span.end()
9. Failure Modes & Prevention
9.1 Hallucination Cascade
Pattern: Agent A hallucinates → Agent B uses hallucinated output → Agent C amplifies → cascading wrong decisions.
import time
class AgentPool:
def __init__(self, idle_timeout: int = 30):
self.agents = {}
self.last_active = {}
self.idle_timeout = idle_timeout
def evict_idle(self):
now = time.time()
for agent_id, last in list(self.last_active.items()):
if now - last > self.idle_timeout:
self.agents.pop(agent_id, None)
self.last_active.pop(agent_id, None)
10.3 Context Reuse
Reuse agent context when same agent handles sequential related tasks — avoids re-loading system prompts and domain context ($0.002-$0.01 saved per handoff).
Disagreement detected (N agents, K opinions)
│
├── N >= 3 and simple majority exists? ──▶ Majority vote → RESOLVED
│
├── N >= 3 but no majority?
│ ├── Weighted vote (seniority x2) → RESOLVED if majority
│ └── Still deadlocked? → Step 3
│
├── N < 3 or deadlocked?
│ ├── Supervisor override → DECIDED (logged as override)
│ └── Budget > threshold? → Human escalation
│
└── Human escalation:
├── Present: options, agent reasoning, vote distribution
└── Human decision → Record for future training
11.5 Decision Tree: Infinite Loop Detection
Handoff A → B initiated
│
├── Check: (A, B) in visited edges?
│ └── YES → HALT. Raise InfiniteLoopError. Log cycle path.
│
├── Check: delegation_depth >= MAX_DEPTH?
│ └── YES → HALT. Raise DelegationDepthExceeded.
│ Escalate to human with full chain.
│
├── Record edge (A, B) in visited set
├── Increment delegation_depth
└── Proceed with handoff
11.6 Decision Tree: Parallel vs Sequential Execution
Task batch received: [T1, T2, T3, T4]
│
├── Dependency graph analysis
│ │
│ ├── T1, T2 are independent? ──▶ Execute in parallel
│ │
│ ├── T3 depends on T1? ──▶ Wait for T1 completion gate
│ │
│ └── T4 depends on T2, T3? ──▶ Wait for both completion gates
│
├── Parallel execution: fan-out to available agents
│ └── Cost: max(individual_latency) + fan_out_overhead
│
└── Sequential execution: chain with state passing
└── Cost: sum(individual_latency) + N * handoff_overhead
12. Ground Rules
#
Negative Constraint
Mechanical Trigger
Violation Response
1
No mutable state passed across agent boundaries
type(state_arg) in (dict, list) detected in handoff
Deep-copy state; pass serialized snapshot with hash
2
No delegation beyond max-depth
delegation_depth >= MAX_DEPTH or cycle detected
HALT chain; escalate to human with full trace
3
No agent output accepted without consistency check
consistency_score < 0.7 between sequential outputs
Inject verification step; re-run with cross-reference
4
No supervisor computation — routing only
supervisor_node.complexity > O(1) or latency > 200ms
Extract computation to worker agent; supervisor routes only
5
No debate without convergence guard
debate_rounds >= max_rounds or delta < threshold for 2 rounds
Reject handoff; replay from last verified checkpoint
7
No idle agents beyond timeout
time.now() - agent.last_active > IDLE_TIMEOUT
Evict agent from pool; re-instantiate if needed
8
No parallel execution of dependent tasks
Dependency graph has edge between tasks in parallel batch
Sequentialize; insert completion gate
13. Gotchas
Supervisor bottleneck ($200K+ in degraded throughput): Single agent routing all tasks hits latency ceiling at ~50 concurrent agents. Mitigation: Partition by domain with multiple supervisors or use hierarchical fan-out.
Checkpoint state drift ($100K+ in inconsistent decisions): 3+ sequential agents mutate shared TypedDict without checkpoint between mutations. Mitigation: Checkpoint after every handoff; verify handoff hash on receipt.
Infinite delegation loop ($50K+ compute waste): Agent A → B → C → A cycle with no detection. Mitigation: visited edge set + delegation_depth counter; halt at depth 5 or cycle.
Debate indefinite refinement ($30K+ token costs): Two agents iteratively "improving" past optimal. Mitigation: max_rounds=5, improvement_threshold=0.05, stagnation detection at 2 rounds.
Hallucination cascade ($500K+ wrong architecture): Agent B acts on Agent A's hallucinated output, compounding error across chain. Mitigation: Inter-agent consistency check with embedding similarity threshold.
Missing human-in-the-loop ($250K+ unauthorized spend): Budget-exceeding decisions auto-executed without escalation. Mitigation: Budget gate before any decision > $threshold; require human approval.
State corruption on concurrent writes ($75K+ data inconsistency): Multiple agents write to shared state without distributed lock. Mitigation: WAL (write-ahead log) or Redis distributed lock on shared state segments.
Context window overflow on long chains ($20K+ truncated decisions): Delegation chain N > 5 agents, each appending to message history → context overflow. Mitigation: Summarize state at each handoff; pass structured state, not raw messages.
14. Quick Start / Implementation Checklist
Phase 1 — Design (Day 1)
Select topology pattern (use Decision Tree 11.1)
Define TypedDict/Pydantic/Message schema for shared state
Create capability manifest for all agents
Set max_depth and timeout thresholds
Phase 2 — Core (Day 2-3)
Implement handoff protocol with cryptographic hash
Implement delegation routing with capability matching
Configure checkpointer (LangGraph) or message-bus (AutoGen)
