Menu
Master QMIX, MADDPG, CTDE - multi-agent learning with coordination and credit assignment
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Baseado na classificação ocupacional SOC
Use when assessing codebase architecture and you feel pressure to soften critique, lead with strengths, or frame problems diplomatically - provides evidence-based critical assessment resisting relationship and economic pressures
Use when cataloging technical debt under time pressure and tempted to explain choices instead of delivering document - enforces execution discipline with scoped delivery patterns for partial catalogs
Use when stakeholders pressure you to change technical priorities and you're tempted to compromise on security-first or call it synthesis - enforces risk-based prioritization over stakeholder preferences
Use when you have architecture documentation from system-archaeologist and need critical assessment, refactoring recommendations, or improvement prioritization - routes to appropriate architect specialist skills
Master A2C, A3C, SAC, TD3 - actor-critic methods for continuous control
Master ε-greedy, UCB, curiosity-driven, RND, intrinsic motivation exploration
| name | multi-agent-rl |
| description | Master QMIX, MADDPG, CTDE - multi-agent learning with coordination and credit assignment |
Invoke this skill when you encounter:
This skill teaches learning from multiple simultaneous agents with coordination challenges.
Do NOT use this skill for:
Multi-agent RL learns coordinated policies for multiple agents in shared environment, solving the fundamental problem that environment non-stationarity from other agents' learning breaks standard RL convergence guarantees.
The core insight: When other agents improve their policies, the environment changes. Your value estimates computed assuming other agents play old policy become wrong when they play new policy.
Single-Agent RL:
1. Agent learns policy π
2. Environment is fixed
3. Agent value estimates Q(s,a) stable
4. Algorithm converges to optimal policy
Multi-Agent RL:
1. Agent 1 learns policy π_1
2. Agent 2 also learning, changing π_2
3. Environment from Agent 1 perspective is non-stationary
4. Agent 1's value estimates invalid when Agent 2 improves
5. Standard convergence guarantees broken
6. Need special algorithms: QMIX, MADDPG, communication
Without addressing non-stationarity, multi-agent learning is unstable.
Without understanding multi-agent problem structure and non-stationarity, you'll implement algorithms that fail to coordinate, suffer credit assignment disasters, or waste effort on agent conflicts instead of collaboration.
Standard RL Assumption (Single-Agent):
Multi-Agent RL Reality:
Cooperative Multi-Agent Problem:
Definition: All agents share same objective
Reward: R_team(a_1, a_2, ..., a_n) = same for all agents
Example - Robot Team Assembly:
- All robots get same team reward
- +100 if assembly succeeds
- 0 if assembly fails
- All robots benefit from success equally
Characteristic:
- Agents don't conflict on goals
- Challenge: Credit assignment (who deserves credit?)
- Solution: Value factorization (QMIX, QPLEX)
Key Insight:
Cooperative doesn't mean agents see each other!
- Agents might have partial/no observation of others
- Still must coordinate for team success
- Factorization enables coordination without observation
Competitive Multi-Agent Problem:
Definition: Agents have opposite objectives (zero-sum)
Reward: R_i(a_1, ..., a_n) = -R_j(a_1, ..., a_n) for i≠j
Example - Chess, Poker, Soccer:
- Agent 1 tries to win
- Agent 2 tries to win
- One's gain is other's loss
- R_1 + R_2 = 0 (zero-sum)
Characteristic:
- Agents are adversarial
- Challenge: Computing best response to opponent
- Solution: Nash equilibrium (MADDPG, self-play)
Key Insight:
In competitive games, agents must predict opponent strategies.
- Agent 1 assumes Agent 2 plays best response
- Agent 2 assumes Agent 1 plays best response
- Nash equilibrium = mutual best response
- No agent can improve unilaterally
Mixed Multi-Agent Problem:
Definition: Some cooperation, some competition
Reward: R_i(a_1, ..., a_n) contains both shared and individual terms
Example - Team Soccer (3v3):
- Blue team agents cooperate for same goal
- But blue vs red is competitive
- Blue agent reward:
R_i = +10 if blue scores, -10 if red scores (team-based)
+ 1 if blue_i scores goal (individual bonus)
Characteristic:
- Agents cooperate with teammates
- Agents compete with opponents
- Challenge: Balancing cooperation and competition
- Solution: Hybrid approaches using both cooperative and competitive algorithms
Key Insight:
Mixed scenarios are most common in practice.
- Robot teams: cooperate internally, compete for resources
- Trading: multiple firms (cooperate via regulations, compete for profit)
- Multiplayer games: team-based (cooperate with allies, compete with enemies)
What is Non-Stationarity?
Stationarity: Environment dynamics P(s'|s,a) and rewards R(s,a) are fixed
Non-Stationarity: Dynamics/rewards change over time
In multi-agent RL:
Environment from Agent 1's perspective:
P(s'_1 | s_1, a_1, a_2(t), a_3(t), ...)
If other agents' policies change:
π_2(t) ≠ π_2(t+1)
Then transition dynamics change:
P(s'_1 | s_1, a_1, a_2(t)) ≠ P(s'_1 | s_1, a_1, a_2(t+1))
Environment is non-stationary!
Why Non-Stationarity Breaks Standard RL:
# Single-agent Q-learning assumes:
# Environment is fixed during learning
# Q-values converge because bellman expectation is fixed point
Q[s,a] ← Q[s,a] + α(r + γ max_a' Q[s',a'] - Q[s,a])
# In multi-agent with non-stationarity:
# Other agents improve their policies
# Max action a' depends on other agents' policies
# When other agents improve, max action changes
# Q-values chase moving target
# No convergence guarantee
Impact on Learning:
Scenario: Two agents learning to navigate
Agent 1 learns: "If Agent 2 goes left, I go right"
Agent 1 builds value estimates based on this assumption
Agent 2 improves: "Actually, going right is better"
Now Agent 2 goes right (not left)
Agent 1's assumptions invalid!
Agent 1's value estimates become wrong
Agent 1 must relearn
Agent 1 tries new path based on new estimates
Agent 2 sees Agent 1's change and adapts
Agent 2's estimates become wrong
Result: Chaotic learning, no convergence
Key Idea: Use centralized information during training, decentralized information during execution.
Training Phase (Centralized):
- Trainer observes: o_1, o_2, ..., o_n (all agents' observations)
- Trainer observes: a_1, a_2, ..., a_n (all agents' actions)
- Trainer observes: R_team or R_1, R_2, ... (reward signals)
- Trainer can assign credit fairly
- Trainer can compute global value functions
Execution Phase (Decentralized):
- Agent 1 observes: o_1 only
- Agent 1 executes: π_1(a_1 | o_1)
- Agent 1 doesn't need to see other agents
- Each agent is independent during rollout
- Enables scalability and robustness
Why CTDE Solves Non-Stationarity:
During training:
- Centralized trainer sees all information
- Can compute value Q_1(s_1, s_2, ..., s_n | a_1, a_2, ..., a_n)
- Can factor: Q_team = f(Q_1, Q_2, ..., Q_n) (QMIX)
- Can compute importance weights: who contributed most?
During execution:
- Decentralized agents only use own observations
- Policies learned during centralized training work well
- No need for other agents' observations at runtime
- Robust to other agents' changes (policy doesn't depend on their states)
Result:
- Training leverages global information for stability
- Execution is independent and scalable
- Solves non-stationarity via centralized credit assignment
Centralized Information Used in Training:
# During training, compute global value function
# Inputs: observations and actions of ALL agents
def compute_value_ctde(obs_1, obs_2, obs_3, act_1, act_2, act_3):
# See everyone's observations
global_state = combine(obs_1, obs_2, obs_3)
# See everyone's actions
joint_action = (act_1, act_2, act_3)
# Compute shared value with all information
Q_shared = centralized_q_network(global_state, joint_action)
# Factor into individual Q-values (QMIX)
Q_1 = q_network_1(obs_1, act_1)
Q_2 = q_network_2(obs_2, act_2)
Q_3 = q_network_3(obs_3, act_3)
# Factorization: Q_team ≈ mixing_network(Q_1, Q_2, Q_3)
# Each agent learns its contribution via QMIX loss
return Q_shared, (Q_1, Q_2, Q_3)
Decentralized Execution:
# During execution, use only own observation
def execute_policy(agent_id, own_observation):
# Agent only sees and uses own obs
action = policy_network(own_observation)
# No access to other agents' observations
# Doesn't need other agents' actions
# Purely decentralized execution
return action
# All agents execute in parallel:
# Agent 1: o_1 → a_1 (decentralized)
# Agent 2: o_2 → a_2 (decentralized)
# Agent 3: o_3 → a_3 (decentralized)
# Execution is independent!
Problem: In cooperative teams, how do you assign credit fairly?
Naive approach: Joint Q-value
Q_team(s, a_1, a_2, ..., a_n) = expected return from joint action
Problem: Still doesn't assign individual credit
If Q_team = 100, how much did Agent 1 contribute?
Agent 1 might think: "I deserve 50%" (overconfident)
But Agent 1 might deserve only 10% (others did more)
Result: Agents learn wrong priorities
Solution: Value Factorization (QMIX)
Key Assumption: Monotonicity in actions
If improving Agent i's action improves team outcome,
improving Agent i's individual Q-value should help
Mathematical Form:
Q_team(a) ≥ Q_team(a') if Agent i plays better action a_i instead of a'_i
and Agent i's Q_i(a_i) > Q_i(a'_i)
Concrete Implementation:
Q_team(s, a_1, ..., a_n) = f(Q_1(s_1, a_1), Q_2(s_2, a_2), ..., Q_n(s_n, a_n))
Where:
- Q_i: Individual Q-network for agent i
- f: Monotonic mixing network (ensures monotonicity)
Monotonicity guarantee:
If Q_1 increases, Q_team increases (if f is monotonic)
Architecture:
Individual Q-Networks: Mixing Network (Monotonic):
┌─────────┐ ┌──────────────────┐
│ Agent 1 │─o_1─────────────────→│ │
│ LSTM │ │ MLP (weights) │─→ Q_team
│ Q_1 │ │ are monotonic │
└─────────┘ │ (ReLU blocks) │
└──────────────────┘
┌─────────┐ ↑
│ Agent 2 │─o_2──────────────────────────┤
│ LSTM │ │
│ Q_2 │ │
└─────────┘ │
Hypernet:
┌─────────┐ generates weights
│ Agent 3 │─o_3────────────────────→ as function
│ LSTM │ of state
│ Q_3 │
└─────────┘
Value outputs: Q_1(o_1, a_1), Q_2(o_2, a_2), Q_3(o_3, a_3)
Mixing: Q_team = mixing_network(Q_1, Q_2, Q_3, state)
QMIX Training:
import torch
import torch.nn as nn
from torch.optim import Adam
class QMIXAgent:
def __init__(self, n_agents, state_dim, obs_dim, action_dim, hidden_dim=64):
self.n_agents = n_agents
# Individual Q-networks (one per agent)
self.q_networks = nn.ModuleList([
nn.Sequential(
nn.Linear(obs_dim + action_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, 1) # Q-value for this action
)
for _ in range(n_agents)
])
# Mixing network: takes individual Q-values and produces joint Q
self.mixing_network = nn.Sequential(
nn.Linear(n_agents + state_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, 1)
)
# Hypernet: generates mixing network weights (ensuring monotonicity)
self.hypernet = nn.Sequential(
nn.Linear(state_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim * (n_agents + state_dim))
)
self.optimizer = Adam(
list(self.q_networks.parameters()) +
list(self.mixing_network.parameters()) +
list(self.hypernet.parameters()),
lr=5e-4
)
self.discount = 0.99
self.target_update_rate = 0.001
self.epsilon = 0.05
# Target networks (soft update)
self._init_target_networks()
def _init_target_networks(self):
"""Create target networks for stable learning."""
self.target_q_networks = nn.ModuleList([
nn.Sequential(*[nn.Linear(*p.shape[::-1]) for p in q.parameters()])
for q in self.q_networks
])
self.target_mixing_network = nn.Sequential(
*[nn.Linear(*p.shape[::-1]) for p in self.mixing_network.parameters()]
)
def compute_individual_q_values(self, observations, actions):
"""
Compute Q-values for each agent given their observation and action.
Args:
observations: list of n_agents observations (each [batch_size, obs_dim])
actions: list of n_agents actions (each [batch_size, action_dim])
Returns:
q_values: tensor [batch_size, n_agents]
"""
q_values = []
for i, (obs, act) in enumerate(zip(observations, actions)):
# Concatenate observation and action
q_input = torch.cat([obs, act], dim=-1)
q_i = self.q_networks[i](q_input)
q_values.append(q_i)
return torch.cat(q_values, dim=-1) # [batch_size, n_agents]
def compute_joint_q_value(self, q_values, state):
"""
Mix individual Q-values into joint Q-value using monotonic mixing network.
Args:
q_values: individual Q-values [batch_size, n_agents]
state: global state [batch_size, state_dim]
Returns:
q_joint: joint Q-value [batch_size, 1]
"""
# Ensure monotonicity by using weight constraints
# Mixing network learns to combine Q-values
q_joint = self.mixing_network(torch.cat([q_values, state], dim=-1))
return q_joint
def train_step(self, batch, state_batch):
"""
One QMIX training step.
Batch contains:
observations: list[n_agents] of [batch_size, obs_dim]
actions: list[n_agents] of [batch_size, action_dim]
rewards: [batch_size] (shared team reward)
next_observations: list[n_agents] of [batch_size, obs_dim]
dones: [batch_size]
"""
observations, actions, rewards, next_observations, dones = batch
batch_size = observations[0].shape[0]
# Compute current Q-values
q_values = self.compute_individual_q_values(observations, actions)
q_joint = self.compute_joint_q_value(q_values, state_batch)
# Compute target Q-values
with torch.no_grad():
# Get next Q-values for all possible joint actions (in practice, greedy)
next_q_values = self.compute_individual_q_values(
next_observations,
[torch.zeros_like(a) for a in actions] # Best actions (simplified)
)
# Mix next Q-values
next_q_joint = self.compute_joint_q_value(next_q_values, state_batch)
# TD target: team gets shared reward
td_target = rewards.unsqueeze(-1) + (
1 - dones.unsqueeze(-1)
) * self.discount * next_q_joint
# QMIX loss
qmix_loss = ((q_joint - td_target) ** 2).mean()
self.optimizer.zero_grad()
qmix_loss.backward()
self.optimizer.step()
# Soft update target networks
self._soft_update_targets()
return {'qmix_loss': qmix_loss.item()}
def _soft_update_targets(self):
"""Soft update target networks."""
for target, main in zip(self.target_q_networks, self.q_networks):
for target_param, main_param in zip(target.parameters(), main.parameters()):
target_param.data.copy_(
self.target_update_rate * main_param.data +
(1 - self.target_update_rate) * target_param.data
)
def select_actions(self, observations):
"""
Greedy action selection (decentralized execution).
Each agent selects action independently.
"""
actions = []
for i, obs in enumerate(observations):
with torch.no_grad():
# Agent i evaluates all possible actions
best_action = None
best_q = -float('inf')
for action in range(self.action_dim):
q_input = torch.cat([obs, one_hot(action, self.action_dim)])
q_val = self.q_networks[i](q_input).item()
if q_val > best_q:
best_q = q_val
best_action = action
# Epsilon-greedy
if torch.rand(1).item() < self.epsilon:
best_action = torch.randint(0, self.action_dim, (1,)).item()
actions.append(best_action)
return actions
QMIX Key Concepts:
When QMIX Works Well:
QMIX Limitations:
Core Idea: Actor-critic but with centralized critic during training.
DDPG (single-agent):
- Actor π(a|s) learns policy
- Critic Q(s,a) estimates value
- Critic trains actor via policy gradient
MADDPG (multi-agent):
- Each agent has actor π_i(a_i|o_i)
- Centralized critic Q(s, a_1, ..., a_n) sees all agents
- During training: use centralized critic for learning
- During execution: each agent uses only own actor
MADDPG Algorithm:
class MADDPGAgent:
def __init__(self, agent_id, n_agents, obs_dim, action_dim, state_dim, hidden_dim=256):
self.agent_id = agent_id
self.n_agents = n_agents
self.action_dim = action_dim
# Actor: learns decentralized policy π_i(a_i|o_i)
self.actor = nn.Sequential(
nn.Linear(obs_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, action_dim),
nn.Tanh() # Continuous actions in [-1, 1]
)
# Critic: centralized value Q(s, a_1, ..., a_n)
# Input: global state + all agents' actions
self.critic = nn.Sequential(
nn.Linear(state_dim + n_agents * action_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, 1) # Single value output
)
# Target networks for stability
self.target_actor = copy.deepcopy(self.actor)
self.target_critic = copy.deepcopy(self.critic)
self.actor_optimizer = Adam(self.actor.parameters(), lr=1e-4)
self.critic_optimizer = Adam(self.critic.parameters(), lr=1e-3)
self.discount = 0.99
self.tau = 0.01 # Soft update rate
def train_step(self, batch):
"""
MADDPG training step.
Batch contains:
observations: list[n_agents] of [batch_size, obs_dim]
actions: list[n_agents] of [batch_size, action_dim]
rewards: [batch_size] (agent-specific reward!)
next_observations: list[n_agents] of [batch_size, obs_dim]
global_state: [batch_size, state_dim]
next_global_state: [batch_size, state_dim]
dones: [batch_size]
"""
observations, actions, rewards, next_observations, \
global_state, next_global_state, dones = batch
batch_size = observations[0].shape[0]
agent_obs = observations[self.agent_id]
agent_action = actions[self.agent_id]
agent_reward = rewards # Agent-specific reward
# Step 1: Critic Update (centralized)
with torch.no_grad():
# Compute next actions using target actors
next_actions = []
for i, next_obs in enumerate(next_observations):
if i == self.agent_id:
next_a = self.target_actor(next_obs)
else:
# Use stored target actors from other agents
next_a = other_agents_target_actors[i](next_obs)
next_actions.append(next_a)
# Concatenate all next actions
next_actions_cat = torch.cat(next_actions, dim=-1)
# Compute next value (centralized critic)
next_q = self.target_critic(
torch.cat([next_global_state, next_actions_cat], dim=-1)
)
# TD target
td_target = agent_reward.unsqueeze(-1) + (
1 - dones.unsqueeze(-1)
) * self.discount * next_q
# Compute current Q-value
current_actions_cat = torch.cat(actions, dim=-1)
current_q = self.critic(
torch.cat([global_state, current_actions_cat], dim=-1)
)
# Critic loss
critic_loss = ((current_q - td_target) ** 2).mean()
self.critic_optimizer.zero_grad()
critic_loss.backward()
self.critic_optimizer.step()
# Step 2: Actor Update (decentralized policy improvement)
# Actor only uses own observation
policy_actions = []
for i, obs in enumerate(observations):
if i == self.agent_id:
# Use current actor for this agent
action_i = self.actor(obs)
else:
# Use current actors of other agents
action_i = other_agents_actors[i](obs)
policy_actions.append(action_i)
# Compute Q-value under current policy
policy_actions_cat = torch.cat(policy_actions, dim=-1)
policy_q = self.critic(
torch.cat([global_state, policy_actions_cat], dim=-1)
)
# Policy gradient: maximize Q-value
actor_loss = -policy_q.mean()
self.actor_optimizer.zero_grad()
actor_loss.backward()
self.actor_optimizer.step()
# Soft update target networks
self._soft_update_targets()
return {
'critic_loss': critic_loss.item(),
'actor_loss': actor_loss.item(),
'avg_q_value': current_q.mean().item()
}
def _soft_update_targets(self):
"""Soft update target networks toward main networks."""
for target_param, main_param in zip(
self.target_actor.parameters(),
self.actor.parameters()
):
target_param.data.copy_(
self.tau * main_param.data + (1 - self.tau) * target_param.data
)
for target_param, main_param in zip(
self.target_critic.parameters(),
self.critic.parameters()
):
target_param.data.copy_(
self.tau * main_param.data + (1 - self.tau) * target_param.data
)
def select_action(self, observation):
"""Decentralized action selection."""
with torch.no_grad():
action = self.actor(observation)
# Add exploration noise
action = action + torch.normal(0, 0.1, action.shape)
action = torch.clamp(action, -1, 1)
return action.cpu().numpy()
MADDPG Key Properties:
When MADDPG Works Well:
Problem Without Communication:
Agents with partial observability:
Agent 1: sees position p_1, but NOT p_2
Agent 2: sees position p_2, but NOT p_1
Goal: Avoid collision while moving to targets
Without communication:
Agent 1: "I don't know where Agent 2 is"
Agent 2: "I don't know where Agent 1 is"
Both might move toward same corridor
Collision, but agents couldn't coordinate!
With communication:
Agent 1: broadcasts "I'm moving left"
Agent 2: receives message, moves right
No collision!
Communication Trade-offs:
Advantages:
- Enables coordination with partial observability
- Can solve some problems impossible without communication
- Explicit intention sharing
Disadvantages:
- Adds complexity: agents must learn what to communicate
- High variance: messages might mislead
- Computational overhead: processing all messages
- Communication bandwidth limited in real systems
When to use communication:
- Partial observability prevents coordination
- Explicit roles (e.g., one agent is "scout")
- Limited field of view, agents are out of sight
- Agents benefit from sharing intentions
When NOT to use communication:
- Full observability (agents see everything)
- Simple coordination (value factorization sufficient)
- Communication is unreliable
Idea: Agents learn to send and receive messages to improve coordination.
Architecture:
1. Each agent processes own observation: f_i(o_i) → hidden state h_i
2. Agent broadcasts hidden state as "message"
3. Agent receives messages from neighbors
4. Agent aggregates messages: Σ_j M(h_j) (attention mechanism)
5. Agent processes aggregated information: policy π(a_i | h_i + aggregated)
Key: Agents learn what information to broadcast in h_i
Receiving agents learn what messages are useful
Simple Communication Example:
class CommNetAgent:
def __init__(self, obs_dim, action_dim, hidden_dim=64):
# Encoding network: observation → hidden message
self.encoder = nn.Sequential(
nn.Linear(obs_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim) # Message to broadcast
)
# Communication aggregation (simplified attention)
self.comm_processor = nn.Sequential(
nn.Linear(hidden_dim * 2, hidden_dim), # Own + received
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim)
)
# Policy network
self.policy = nn.Sequential(
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, action_dim)
)
def compute_message(self, observation):
"""Generate message to broadcast to other agents."""
return self.encoder(observation)
def forward(self, observation, received_messages):
"""
Process observation + received messages, output action.
Args:
observation: [obs_dim]
received_messages: list of messages from neighbors
Returns:
action: [action_dim]
"""
# Generate own message
my_message = self.encoder(observation)
# Aggregate received messages (mean pooling)
if received_messages:
others_messages = torch.stack(received_messages).mean(dim=0)
else:
others_messages = torch.zeros_like(my_message)
# Process aggregated communication
combined = torch.cat([my_message, others_messages], dim=-1)
hidden = self.comm_processor(combined)
# Select action
action = self.policy(hidden)
return action, my_message
Communication Pitfall: Agents learn to send misleading messages!
# Without careful design, agents learn deceptive communication:
Agent 1 learns: "If I broadcast 'I'm going right', Agent 2 will go left"
Agent 1 broadcasts: "Going right" (but actually goes left)
Agent 2 goes right as expected (collision!)
Agent 1 gets higher reward (its deception worked)
Solution: Design communication carefully
- Verify agents to be truthful (implicit in cooperative setting)
- Use communication only when beneficial
- Monitor emergent communication protocols
Problem:
Scenario: 3-robot assembly team
Team reward: +100 if assembly succeeds, 0 if fails
Individual Reward Design:
Option 1 - Split equally: each robot gets +33.33
Problem: Robot 3 (insignificant) gets same credit as Robot 1 (crucial)
Option 2 - Use agent contribution:
Robot 1 (held piece): +60
Robot 2 (guided insertion): +25
Robot 3 (steadied base): +15
Problem: How to compute contributions? (requires complex analysis)
Option 3 - Use value factorization (QMIX):
Team value = mixing_network(Q_1, Q_2, Q_3)
Each robot learns its Q-value
QMIX learns to weight Q-values by importance
Result: Fair credit assignment via factorization
QMIX Credit Assignment Mechanism:
Training:
Observe: robot_1 does action a_1, gets q_1
robot_2 does action a_2, gets q_2
robot_3 does action a_3, gets q_3
Team gets reward r_team
Factorize: r_team ≈ mixing_network(q_1, q_2, q_3)
= w_1 * q_1 + w_2 * q_2 + w_3 * q_3 + bias
Learn weights w_i via mixing network
If Robot 1 is crucial:
mixing network learns w_1 > w_2, w_3
Robot 1 gets larger credit (w_1 * q_1 > others)
If Robot 3 is redundant:
mixing network learns w_3 ≈ 0
Robot 3 gets small credit
Result: Each robot learns fair contribution
Value Decomposition Pitfall: Agents can game the factorization!
Example: Learned mixing network w = [0.9, 0.05, 0.05]
Agent 1 learns: "I must maximize q_1 (it has weight 0.9)"
Agent 1 tries: action that maximizes own q_1
Problem: q_1 computed from own reward signal (myopic)
might not actually help team!
Solution: Use proper credit assignment metrics
- Shapley values: game theory approach to credit
- Counterfactual reasoning: what if agent didn't act?
- Implicit credit (QMIX): let factorization emergently learn
Symptom: Learning curves erratic, no convergence.
# Problem scenario:
for episode in range(1000):
# Agent 1 learns
episode_reward_1 = []
for t in range(steps):
a_1 = agent_1.select_action(o_1)
a_2 = agent_2.select_action(o_2) # Using old policy!
r, o'_1, o'_2 = env.step(a_1, a_2)
agent_1.update(a_1, r, o'_1)
# Agent 2 improves (environment changes for Agent 1!)
episode_reward_2 = []
for t in range(steps):
a_1 = agent_1.select_action(o_1) # OLD VALUE ESTIMATES
a_2 = agent_2.select_action(o_2) # NEW POLICY (Agent 2 improved)
r, o'_1, o'_2 = env.step(a_1, a_2)
agent_2.update(a_2, r, o'_2)
Result: Agent 1's Q-values become invalid when Agent 2 improves
Learning is unstable, doesn't converge
Solution: Use CTDE or opponent modeling
# CTDE Approach:
# During training, use global information to stabilize
trainer.observe(o_1, a_1, o_2, a_2, r)
# Trainer sees both agents' actions, can compute stable target
# During execution:
agent_1.execute(o_1 only) # Decentralized
agent_2.execute(o_2 only) # Decentralized
Symptom: Agents don't improve, stuck at local optima.
# Problem: Multi-agent team, shared reward
total_reward = 50
# Distribution: who gets what?
# Agent 1 thinks: "I deserve 50" (overconfident)
# Agent 2 thinks: "I deserve 50" (overconfident)
# Agent 3 thinks: "I deserve 50" (overconfident)
# Each agent overestimates importance
# Each agent learns selfishly (internal conflict)
# Team coordination breaks
Result: Team performance worse than if agents cooperated
Solution: Use value factorization
# QMIX learns fair decomposition
q_1, q_2, q_3 = compute_individual_values(a_1, a_2, a_3)
team_reward = mixing_network(q_1, q_2, q_3)
# Mixing network learns importance
# If Agent 2 crucial: weight_2 > weight_1, weight_3
# Training adjusts weights based on who actually helped
Result: Fair credit, agents coordinate
Symptom: Learning fails in specific problem types (cooperative/competitive).
# Problem: Using QMIX (cooperative) in competitive setting
# Competitive game (agents have opposite rewards)
# QMIX assumes: shared reward (monotonicity works)
# But in competitive:
# Q_1 high means Agent 1 winning
# Q_2 high means Agent 2 winning (opposite!)
# QMIX mixing doesn't make sense
# Convergence fails
# Solution: Use MADDPG (handles competitive)
# MADDPG doesn't assume monotonicity
# Works with individual rewards
# Handles competition naturally
Use MARL when:
1. Multiple simultaneous learners
- Problem has 2+ agents learning
- NOT just parallel tasks (that's single-agent x N)
2. Shared/interdependent environment
- Agents' actions affect each other
- One agent's action impacts other agent's rewards
- True interaction (not independent MDPs)
3. Coordination is beneficial
- Agents can improve by coordinating
- Alternative: agents could act independently (inefficient)
4. Non-trivial communication/credit
- Agents need to coordinate or assign credit
- NOT trivial to decompose into independent subproblems
Use Single-Agent RL when:
1. Single learning agent (others are environment)
- Example: one RL agent vs static rules-based opponents
- Environment includes other agents, but they're not learning
2. Independent parallel tasks
- Example: 10 robots, each with own goal, no interaction
- Use single-agent RL x 10 (faster, simpler)
3. Fully decomposable problems
- Example: multi-robot path planning (can use single-agent per robot)
- Problem decomposes into independent subproblems
4. Scalability critical
- Single-agent RL scales to huge teams
- MARL harder to scale (centralized training bottleneck)
Problem: Multiple agents learning together?
NO → Use single-agent RL
YES ↓
Problem: Agents' rewards interdependent?
NO → Use single-agent RL x N (parallel)
YES ↓
Problem: Agents must coordinate?
NO → Use independent learning (but expect instability)
YES ↓
Problem structure:
COOPERATIVE → Use QMIX, MAPPO, QPLEX
COMPETITIVE → Use MADDPG, self-play
MIXED → Use hybrid (cooperative + competitive algorithms)
Problem Without Opponent Modeling:
Agent 1 (using MADDPG) learns:
"Move right gives Q=50"
But assumption: Agent 2 plays policy π_2
When Agent 2 improves to π'_2:
"Move right gives Q=20" (because Agent 2 blocks that path)
Agent 1's Q-value estimates become stale!
Environment has changed (opponent improved)
Solution: Opponent Modeling
class OpponentModelingAgent:
def __init__(self, agent_id, n_agents, obs_dim, action_dim):
self.agent_id = agent_id
# Own actor and critic
self.actor = self._build_actor(obs_dim, action_dim)
self.critic = self._build_critic()
# Model opponent policies (for agents we compete against)
self.opponent_models = {
i: self._build_opponent_model() for i in range(n_agents) if i != agent_id
}
def _build_opponent_model(self):
"""Model what opponent will do given state."""
return nn.Sequential(
nn.Linear(64, 128),
nn.ReLU(),
nn.Linear(128, 128),
nn.ReLU(),
nn.Linear(128, self.action_dim)
)
def train_step_with_opponent_modeling(self, batch):
"""
Update own policy AND opponent models.
Key insight: predict what opponent will do,
then plan against those predictions
"""
observations, actions, rewards, next_observations = batch
# Step 1: Update opponent models (supervised)
# Predict opponent action from observation
for opponent_id, model in self.opponent_models.items():
predicted_action = model(next_observations[opponent_id])
actual_action = actions[opponent_id]
opponent_loss = ((predicted_action - actual_action) ** 2).mean()
# Update opponent model
optimizer.zero_grad()
opponent_loss.backward()
optimizer.step()
# Step 2: Plan against opponent predictions
predicted_opponent_actions = {
i: model(observations[i])
for i, model in self.opponent_models.items()
}
# Use predictions in MADDPG update
# Critic sees: own obs + predicted opponent actions
# Actor learns: given opponent predictions, best response
return {'opponent_loss': opponent_loss.item()}
Opponent Modeling Trade-offs:
Advantages:
- Accounts for opponent improvements (non-stationarity)
- Enables planning ahead
- Reduces brittleness to opponent policy changes
Disadvantages:
- Requires learning opponent models (additional supervision)
- If opponent model is wrong, agent learns wrong policy
- Computational overhead
- Assumes opponent is predictable
When to use:
- Competitive settings with clear opponents
- Limited number of distinct opponents
- Opponents have consistent strategies
When NOT to use:
- Too many potential opponents
- Opponents are unpredictable
- Cooperative setting (waste of resources)
Idea: Each agent learns Q-value using only own rewards and observations.
class IQLMultiAgent:
def __init__(self, agent_id, obs_dim, action_dim):
self.agent_id = agent_id
# Q-network for this agent only
self.q_network = nn.Sequential(
nn.Linear(obs_dim + action_dim, 128),
nn.ReLU(),
nn.Linear(128, 1)
)
self.optimizer = Adam(self.q_network.parameters(), lr=1e-3)
def train_step(self, batch):
"""
Independent Q-learning: each agent learns from own reward only.
Problem: Non-stationarity
- Other agents improve policies
- Environment from this agent's perspective changes
- Q-values become invalid
Benefit: Decentralized
- No centralized training needed
- Scalable to many agents
"""
observations, actions, rewards, next_observations = batch
# Q-value update (standard Q-learning)
with torch.no_grad():
# Greedy next action (assume agent acts greedily)
next_q_values = []
for action in range(self.action_dim):
q_input = torch.cat([next_observations, one_hot(action)])
q_val = self.q_network(q_input)
next_q_values.append(q_val)
max_next_q = torch.max(torch.stack(next_q_values), dim=0)[0]
td_target = rewards + 0.99 * max_next_q
# Current Q-value
q_pred = self.q_network(torch.cat([observations, actions], dim=-1))
# TD loss
loss = ((q_pred - td_target) ** 2).mean()
self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()
return {'loss': loss.item()}
IQL in Multi-Agent: Pros and Cons:
Advantages:
- Fully decentralized (scalable)
- No communication needed
- Simple implementation
- Works with partial observability
Disadvantages:
- Non-stationarity breaks convergence
- Agents chase moving targets (other agents improving)
- No explicit coordination
- Performance often poor without CTDE
Result:
- IQL works but is unstable in true multi-agent settings
- Better to use CTDE (QMIX, MADDPG) for stability
- IQL useful if centralized training impossible
Problem:
In single-agent RL:
Experience replay stores (s, a, r, s', d)
Sample uniformly from buffer
Works well (iid samples)
In multi-agent RL:
Experience replay stores (s, a_1, a_2, ..., a_n, r, s')
But agents are non-stationary!
Transition (s, a_1, a_2, r, s') valid only if:
- Assumptions about other agents' policies still hold
- If other agents improved, assumptions invalid
Solution: Prioritized experience replay for multi-agent
- Prioritize transitions where agent's assumptions are likely correct
- Down-weight transitions from old policies (outdated assumptions)
- Focus on recent transitions (more relevant)
Batch Sampling Strategy:
class MultiAgentReplayBuffer:
def __init__(self, capacity=100000, n_agents=3):
self.buffer = deque(maxlen=capacity)
self.n_agents = n_agents
self.priority_weights = deque(maxlen=capacity)
def add(self, transition):
"""Store experience with priority."""
# transition: (observations, actions, rewards, next_observations, dones)
self.buffer.append(transition)
# Priority: how relevant is this to current policy?
# Recent transitions: high priority (policies haven't changed much)
# Old transitions: low priority (agents have improved, assumptions stale)
priority = self._compute_priority(transition)
self.priority_weights.append(priority)
def _compute_priority(self, transition):
"""Compute priority for multi-agent setting."""
# Heuristic: prioritize recent transitions
# Could use TD-error (how surprised are we by this transition?)
age = len(self.buffer) # How long ago was this added?
decay = 0.99 ** age # Exponential decay
return decay
def sample(self, batch_size):
"""Sample prioritized batch."""
# Weighted sampling: high priority more likely
indices = np.random.choice(
len(self.buffer),
batch_size,
p=self.priority_weights / self.priority_weights.sum()
)
batch = [self.buffer[i] for i in indices]
return batch
Users often rationalize MARL mistakes:
Cooperative teams with policy gradients:
class MAPPOAgent:
def __init__(self, agent_id, obs_dim, action_dim, hidden_dim=256):
self.agent_id = agent_id
# Actor: policy for decentralized execution
self.actor = nn.Sequential(
nn.Linear(obs_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, action_dim)
)
# Critic: centralized value function (uses global state during training)
self.critic = nn.Sequential(
nn.Linear(obs_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, 1)
)
self.actor_optimizer = Adam(self.actor.parameters(), lr=3e-4)
self.critic_optimizer = Adam(self.critic.parameters(), lr=1e-3)
def train_step_on_batch(self, observations, actions, returns, advantages):
"""
MAPPO training: advantage actor-critic with clipped policy gradient.
Key difference from DDPG:
- Policy gradient (not off-policy value)
- Centralized training (uses global returns/advantages)
- Decentralized execution (policy uses only own observation)
"""
# Actor loss (clipped PPO)
action_probs = torch.softmax(self.actor(observations), dim=-1)
action_log_probs = torch.log(action_probs.gather(-1, actions))
# Importance weight (in on-policy setting, = 1)
# In practice, small advantage clipping for stability
policy_loss = -(action_log_probs * advantages).mean()
# Entropy regularization (exploration)
entropy = -(action_probs * torch.log(action_probs + 1e-8)).sum(dim=-1).mean()
actor_loss = policy_loss - 0.01 * entropy
self.actor_optimizer.zero_grad()
actor_loss.backward()
self.actor_optimizer.step()
# Critic loss (value estimation)
values = self.critic(observations)
critic_loss = ((values - returns) ** 2).mean()
self.critic_optimizer.zero_grad()
critic_loss.backward()
self.critic_optimizer.step()
return {
'actor_loss': actor_loss.item(),
'critic_loss': critic_loss.item(),
'entropy': entropy.item()
}
MAPPO vs QMIX:
QMIX:
- Value-based (discrete actions)
- Value factorization (credit assignment)
- Works with partial observability
MAPPO:
- Policy gradient-based
- Centralized critic (advantage estimation)
- On-policy (requires recent trajectories)
Use MAPPO when:
- Continuous or large discrete action spaces
- On-policy learning acceptable
- Value factorization not needed (reward structure simple)
Use QMIX when:
- Discrete actions
- Need explicit credit assignment
- Off-policy learning preferred
Problem: Training competitive agents requires opponents.
Naive approach:
- Agent 1 trains vs fixed opponent
- Problem: fixed opponent doesn't adapt
- Agent 1 learns exploitation (brittle to new opponents)
Self-play:
- Agent 1 trains vs historical versions of itself
- Agent 1 improves → creates stronger opponent
- New Agent 1 trains vs stronger Agent 1
- Cycle: both improve together
- Result: robust agent that beats all versions of itself
Self-Play Implementation:
class SelfPlayTrainer:
def __init__(self, agent_class, n_checkpoint_opponents=5):
self.current_agent = agent_class()
self.opponent_pool = [] # Keep historical versions
self.n_checkpoints = n_checkpoint_opponents
def train(self, num_episodes):
"""Train with self-play against previous versions."""
for episode in range(num_episodes):
# Select opponent: current agent or historical version
if not self.opponent_pool or np.random.rand() < 0.5:
opponent = copy.deepcopy(self.current_agent)
else:
opponent = np.random.choice(self.opponent_pool)
# Play episode: current_agent vs opponent
trajectory = self._play_episode(self.current_agent, opponent)
# Train current agent on trajectory
self.current_agent.train_on_trajectory(trajectory)
# Periodically add current agent to opponent pool
if episode % (num_episodes // self.n_checkpoints) == 0:
self.opponent_pool.append(copy.deepcopy(self.current_agent))
return self.current_agent
def _play_episode(self, agent1, agent2):
"""Play episode: agent1 vs agent2, collect experience."""
trajectory = []
state = self.env.reset()
done = False
while not done:
# Agent 1 action
action1 = agent1.select_action(state['agent1_obs'])
# Agent 2 action (opponent)
action2 = agent2.select_action(state['agent2_obs'])
# Step environment
state, reward, done = self.env.step(action1, action2)
trajectory.append({
'obs1': state['agent1_obs'],
'obs2': state['agent2_obs'],
'action1': action1,
'action2': action2,
'reward1': reward['agent1'],
'reward2': reward['agent2']
})
return trajectory
Self-Play Benefits and Pitfalls:
Benefits:
- Agents automatically improve together
- Robust to different opponent styles
- Emergent complexity (rock-paper-scissors dynamics)
Pitfalls:
- Agents might exploit specific weaknesses (not generalizable)
- Training unstable if pool too small
- Forgetting how to beat weaker opponents (catastrophic forgetting)
- Computational cost (need to evaluate multiple opponents)
Solution: Diverse opponent pool
- Keep varied historical versions
- Mix self-play with evaluation vs fixed benchmark
- Monitor for forgetting (test vs all opponents periodically)
Key consideration: Partial vs full observability
# Full Observability (not realistic but simplest)
observation = {
'own_position': agent_pos,
'all_agent_positions': [pos1, pos2, pos3], # See everyone!
'all_agent_velocities': [vel1, vel2, vel3],
'targets': [target1, target2, target3]
}
# Partial Observability (more realistic, harder)
observation = {
'own_position': agent_pos,
'own_velocity': agent_vel,
'target': own_target,
'nearby_agents': agents_within_5m, # Limited field of view
# Note: don't see agents far away
}
# Consequence: With partial obs, agents must communicate or learn implicitly
# Via environmental interaction (e.g., bumping into others)
Critical for multi-agent learning:
# Cooperative game: shared reward
team_reward = +100 if goal_reached else 0
# Problem: ambiguous who contributed
# Cooperative game: mixed rewards (shared + individual)
team_reward = +100 if goal_reached
individual_bonus = +5 if agent_i_did_critical_action
total_reward_i = team_reward + individual_bonus # incentivizes both
# Competitive game: zero-sum
reward_1 = goals_1 - goals_2
reward_2 = goals_2 - goals_1 # Opposite
# Competitive game: individual scores
reward_1 = goals_1
reward_2 = goals_2
# Problem: agents don't care about each other (no implicit competition)
# Mixed: cooperation + competition (team sports)
reward_i = +10 if team_wins
+ 1 if agent_i_scores
+ 0.1 * team_score # Shared team success bonus
Reward Design Pitfall: Too much individual reward breaks cooperation
Example: 3v3 soccer
reward_i = +100 if agent_i_scores (individual goal)
+ +5 if agent_i_assists (passes to scorer)
+ 0 if teammate scores (not rewarded!)
Result:
Agent learns: "Only my goals matter, don't pass to teammates"
Agent hoards ball, tries solo shots
Team coordination breaks
Lose to coordinated opponent team
Solution: Include team reward
reward_i = +100 if team_wins
+ +10 if agent_i_scores goal
+ +2 if agent_i_assists
Multi-agent RL is needed when:
Multi-agent RL is NOT needed when:
Key Algorithms:
Algorithm Selection Matrix:
Cooperative Competitive Mixed
Discrete Action QMIX Nash-Q Hybrid
Continuous Action MAPPO/MADDPG MADDPG MADDPG
Partial Obs +Comm +Opponent Mod +Both
Scalable IQL (unstable) IQL IQL (unstable)
Critical Success Factors:
Use this skill to understand multi-agent problem structure and select appropriate algorithms for coordination challenges.