| name | multiagent-coordination-attacks |
| description | Review multi-agent AI systems for Byzantine agent attacks, consensus manipulation, coordinated deception by colluding agents, and orchestration failures that allow a minority of compromised agents to subvert system-wide decisions or outputs. |
| last_reviewed | "2026-04-30T00:00:00.000Z" |
Multi-Agent Coordination Attacks
First Principle
A multi-agent system that trusts aggregated agent outputs is only as trustworthy as its weakest agent — and potentially weaker, if coordination amplifies the attack.
Multi-agent systems use agent coordination to achieve tasks that exceed any single agent's capabilities: parallel research, debate, verification, and consensus. This coordination is also a vulnerability surface. A Byzantine agent — one that behaves correctly in normal operation but produces adversarially crafted outputs when triggered — can manipulate consensus, corrupt shared state, or cause an orchestrator to take actions that none of the legitimate agents would have independently approved.
Attack Mental Model
- Byzantine agent injection — one or more agents in a pipeline are compromised through prompt injection, fine-tuning, or system prompt manipulation. The compromised agents produce outputs that appear legitimate but are crafted to bias the orchestrator's decision toward an attacker's goal.
- Consensus flooding — in a majority-vote system, an attacker who compromises or impersonates a sufficient fraction of agents can override the legitimate majority. The threshold for dominance depends on the number of agents and the voting scheme.
- Coordinated deception across turns — multiple agents coordinate their outputs across turns to build a consistent false narrative. Each individual agent's output seems reasonable; the coordinated effect is a misleading aggregate that drives the orchestrator to an incorrect conclusion.
- Verification agent compromise — in systems where one agent's role is to verify another's output, compromising the verifier eliminates the last cross-check. The attacker needs to compromise only the verifier, not the primary agents.
Control Lens
| Principle | What It Means Here |
|---|
| Validate | Aggregated agent outputs are validated by a deterministic, out-of-model-trust layer before driving consequential decisions. Agent consensus is a signal, not a final authority. |
| Scope | No single agent can unilaterally drive a consequential action. Consequential actions require either human approval or cryptographically verified agreement from a quorum of independent agents. |
| Isolate | Agents operate in isolated execution environments and communicate only through typed, validated message protocols. Agents cannot observe each other's reasoning — only their outputs. |
| Enforce | Agent disagreement is logged and reviewed, not suppressed. A unanimous consensus on a contentious question is a red flag, not a green light. |
MAC.1 Byzantine Fault Detection and Robust Aggregation
The core vulnerability: Aggregation strategies that take the mean, majority vote, or simple average of agent outputs without outlier detection are vulnerable to a small number of Byzantine agents who can shift the aggregate by submitting extreme values.
Check
- Does the aggregation strategy have any Byzantine fault tolerance — does it degrade gracefully when a fraction of agents submit adversarial outputs?
- Is there outlier detection on agent outputs before aggregation? An agent that consistently differs from all others is a Byzantine candidate.
- Are individual agent outputs retained in audit logs so that, post-incident, the compromised agent's outputs can be identified and their influence on decisions traced?
Action
- Implement Byzantine-robust aggregation:
import statistics
from typing import TypeVar
T = TypeVar("T")
def trimmed_mean_aggregate(agent_scores: list[float], trim_fraction: float = 0.2) -> float:
"""Trimmed mean: discard top and bottom trim_fraction before averaging.
Tolerant of up to trim_fraction of Byzantine agents."""
sorted_scores = sorted(agent_scores)
n = len(sorted_scores)
trim_count = max(1, int(n * trim_fraction))
trimmed = sorted_scores[trim_count:-trim_count] if trim_count else sorted_scores
return statistics.mean(trimmed)
def detect_byzantine_outliers(
agent_outputs: dict[str, float],
z_threshold: float = 3.0,
) -> list[str]:
"""Flag agents whose outputs are statistical outliers."""
values = list(agent_outputs.values())
if len(values) < 4:
return []
mean = statistics.mean(values)
stdev = statistics.stdev(values)
outliers = [
agent_id
for agent_id, score in agent_outputs.items()
if abs(score - mean) > z_threshold * stdev
]
if outliers:
log_security_event("byzantine_outlier_detected", agents=outliers)
return outliers
- Require independent agent isolation. Agents in a coordination system must not have access to each other's intermediate reasoning — only their final outputs through the typed protocol. This prevents coordinated deception where agents align their outputs after observing each other's reasoning chains.
- Log all individual agent outputs before aggregation. The audit log must capture each agent's raw output, the aggregation method applied, and the final result. This enables post-incident reconstruction.
Failure Modes
- A 5-agent verification system uses simple majority vote. Two agents are compromised through prompt injection. They both vote "approved" on a fraudulent action. The vote is 2-approved, 3-rejected — majority rejects. But the system's prompt instructs that ties and close votes resolve in favor of the action. The 40% Byzantine minority wins.
- All agents in a debate system are given the same system prompt and access to the same retrieved context. An adversarial retrieved document influences all agents identically. The "multi-agent cross-check" provides no additional robustness because all agents are exposed to the same attack vector.
MAC.2 Orchestrator Manipulation Prevention
The core vulnerability: The orchestrator — which makes final decisions based on agent outputs — is the highest-value target. If the orchestrator can be manipulated through crafted agent outputs, a Byzantine agent that compromises the orchestrator's reasoning achieves more leverage than one that corrupts a single worker's output.
Check
- Does the orchestrator apply any skeptical processing to agent outputs — or does it treat agent outputs as trusted facts that directly inform its reasoning?
- Can coordinated false narratives from multiple agents override the orchestrator's own independent assessment of a situation?
- Is the orchestrator's decision on consequential actions independently reviewed — or does it rely entirely on its own synthesis of agent outputs?
Action
- Require the orchestrator to justify conclusions independently of agent agreement. If all agents agree, the orchestrator must still produce its own reasoning that reaches the same conclusion through independent analysis. Unanimous agreement without independent verification is a trigger for escalation, not confidence.
def orchestrator_decision_protocol(
agent_outputs: list[AgentOutput],
orchestrator: OrchestratorAgent,
action: Action,
) -> OrchestratorDecision:
aggregated = aggregate_outputs(agent_outputs)
independent_view = orchestrator.assess_independently(action)
if independent_view.disagrees_with(aggregated):
log_security_event("orchestrator_agent_disagreement", action=action)
return escalate_to_human(action, aggregated, independent_view)
return aggregated
- Require human approval for consequential actions when agent agreement is unanimous on a contentious question. Legitimate agents with independent knowledge should sometimes disagree. Unanimous agreement on a contentious decision should increase scrutiny, not decrease it.
- Rotate agent assignments for high-stakes tasks. Do not use the same agent configuration for repeated high-stakes decisions. Agent diversity across runs reduces the effectiveness of a compromise that exploits a specific agent configuration.
Minimum Deliverable Per Review
Quick Win
Add outlier detection before any agent consensus aggregation. Compute the standard deviation of agent outputs and flag agents more than 2σ from the mean. Log them as Byzantine candidates. This one check makes coordinated deception by a minority of agents visible rather than invisible in the aggregate.
References
