formulate-marl-architecture

name	Formulate MARL Architecture
description	This skill should be used when the user asks to "formulate multi-agent RL", "design MARL architecture", "implement CTDE", "set up cooperative or competitive agents", "design MAPPO", "implement QMIX", "handle multi-agent observation spaces", or "define agent communication protocols". Do NOT hallucinate parameters outside the boundaries of Formulate MARL Architecture.
version	0.1.0

Formulate MARL Architecture

Formulate Multi-Agent RL mapping including paradigm selection (cooperative, competitive, mixed), execution strategy (CTDE vs independent), and communication protocol design. This skill produces a formal MARL specification.

Step 1 — Establish the Objective Paradigm

Classify the agent relationships before choosing any algorithm:

Paradigm	Reward Structure	Examples	Recommended Algorithm Family
Fully Cooperative	All agents share one global reward $r_{\text{team}}$	Search-and-rescue robots, football team	MAPPO, QMIX, MADDPG
Competitive (Zero-Sum)	$r_A = -r_B$	Chess, Starcraft 1v1, trading markets	Self-play + PPO, AlphaStar
Mixed Motive	Agents have personal objectives but require occasional cooperation	Traffic navigation, resource allocation	MAPPO with individual rewards + team bonus

Step 2 — Architect the MARL Execution Strategy

Option A — Independent Learning (Naive baseline): Each agent treats other agents as part of the environment. Uses standard single-agent algorithms (PPO, DQN) with no architectural changes.

Observation:  O_i (local)
Policy:       π_i(a_i | O_i)
Critic:       V_i(O_i) or Q_i(O_i, a_i)

Risk: Non-convergence due to non-stationarity — each agent's policy is changing, making the environment appear non-Markovian to others.

Option B — CTDE (Centralized Training, Decentralized Execution): The gold standard for cooperative MARL. During training only, the critic has access to the global state $S$ and all agents' actions ${a_1, ..., a_N}$. During execution, each actor uses only local observations.

Training:
  Global Critic:  V(S, a_1, ..., a_N)  — sees everything
  Local Actors:   π_i(a_i | O_i)       — sees only local obs

Execution:
  Only local actors deployed — no global information required

CTDE implementations: MAPPO (shared or separate actors + centralized critic), MADDPG (continuous), QMIX (monotonic value factorization for cooperative).

Step 3 — Define Observation Separation

Specify exactly what each component can observe:

Component	Observes	Cannot Observe
Local Actor $i$	$O_i$: agent $i$'s local sensors	Other agents' private states, global state
Global Critic (training only)	Full state $S$ = concatenation of all $O_i$ + global context	(Nothing hidden during training)

Define the observation vector for each agent explicitly. Agents with homogeneous observation spaces can share policy weights (parameter sharing) — a major sample efficiency gain.

Parameter sharing rule: Use shared weights if agents have identical roles. Use separate networks if agents have asymmetric roles (e.g., attacker vs. defender).

Step 4 — Establish Communication Protocols

If agents communicate explicitly:

Communication Type	Architecture	When to Use
No communication	Independent actors	Execution bandwidth limited, simple tasks
Broadcast message	Each agent broadcasts a fixed-dim tensor to all others	Cooperative tasks where global coordination helps
Graph-based	Graph Neural Network (GNN) with agent nodes and edge messages	Dynamic agent counts, spatial proximity graphs
Attention-based	Transformer self-attention over agent tokens	Variable number of agents, rich coordination needs

Communication only occurs during execution if bandwidth allows. Training-time communication via the global critic is always free.

Output Format

When utilizing this skill, output:

MARL Paradigm: [Cooperative / Competitive (Zero-Sum) / Mixed Motive]

Value Estimation Model: [CTDE (MAPPO) / Independent Multi-agent PPO / QMIX / MADDPG]

Observation Separation:

• Global Critic sees: [Description of global state $S$]
• Local Actor $i$ sees: [Description of local observation $O_i$]

Parameter Sharing: [Yes — identical roles / No — asymmetric roles]

Communication Protocol: [None / Broadcast / GNN / Attention — with architecture spec]

Additional Resources

Reference Files

• references/marl-algorithms.md — MAPPO, QMIX, MADDPG, HAPPO algorithm details, non-stationarity analysis, credit assignment methods

Scripts

• scripts/validate_marl_spec.py — Validates a MARL specification dict for consistency between paradigm, algorithm choice, and observation separation