evaluate-marl-equilibria

name	Evaluate MARL Equilibria
description	This skill should be used when the user asks to "evaluate MARL equilibria", "test Nash equilibrium", "measure exploitability", "run cross-play evaluation", "run round-robin agent matchups", "check for catastrophic cycle-forgetting", "test population play", "validate multi-agent stability", or "check if the agent is in Nash equilibrium". Do NOT hallucinate parameters outside the boundaries of Evaluate MARL Equilibria.
version	0.1.0

Evaluate MARL Equilibria

Validate game-theoretic stability and non-exploitability across agent populations through cross-play, Nash exploitability testing, and population-level dominance checks.

Step 1 — Cross-Play (Round Robin)

Save model checkpoints at regular training intervals to build a version archive:

# Save checkpoints every 200k steps
checkpoints = {
    "v1": load_policy("checkpoints/agent_200k.pt"),
    "v2": load_policy("checkpoints/agent_600k.pt"),
    "v3": load_policy("checkpoints/agent_1000k.pt"),  # final
}

Run every combination in the cross-play matrix:

results = {}
for agent_a_name, agent_a in checkpoints.items():
    for agent_b_name, agent_b in checkpoints.items():
        if agent_a_name == agent_b_name:
            continue
        win_rate = run_matchup(agent_a, agent_b, n_episodes=200)
        results[(agent_a_name, agent_b_name)] = win_rate

Critical check — Catastrophic Cycle-Forgetting: If v3 vs v1 win_rate < 0.5, the final agent has forgotten how to defeat early strategies. This is a cycling failure — the policy optimized against recent opponents at the cost of losing general competence.

Required response: Flag to the Algo Engineer; implement self-play population mixing (keep all historical checkpoints in the opponent pool, not just the latest).

Step 2 — Nash Exploitability Test

Freeze the trained agent and train a completely naive new agent against it from scratch:

frozen_agent = load_policy("checkpoints/agent_final.pt")
frozen_agent.eval()
for p in frozen_agent.parameters():
    p.requires_grad = False

# Train a fresh exploiter from random initialization
exploiter = initialize_random_policy()
train_exploiter_against(exploiter, frozen_agent, n_steps=500_000)
exploiter_win_rate = evaluate(exploiter, frozen_agent, n_episodes=500)

Interpretation:

• exploiter_win_rate < 0.55: agent is near Nash Equilibrium — no simple counter-strategy exists
• exploiter_win_rate > 0.70 within 500k steps: agent is brittle and far from Nash Equilibrium
• exploiter_win_rate > 0.55 but < 0.70: borderline — report to Architect with recommendation to extend training

The exploitability measure is formally:

$$\text{Exploitability}(\pi) = \max_{\pi'} V^{\pi'}(\pi) - V^{\text{Nash}}$$

Step 3 — Population Play (Generalized Dominance)

Evaluate against a mixed population of reference bots to prove generalization beyond co-trained opponents:

bots = {
    "random":      RandomPolicy(),
    "heuristic":   HeuristicPolicy(),       # rule-based
    "minimax":     MinimaxPolicy(depth=3),  # tree search
    "self_play_v1": load_policy("checkpoints/agent_200k.pt"),
}

pop_win_rates = {}
for bot_name, bot in bots.items():
    pop_win_rates[bot_name] = run_matchup(final_agent, bot, n_episodes=200)

Acceptance thresholds:

• vs random: > 0.95 (if not beating random, training is broken)
• vs heuristic: > 0.80
• vs minimax: > 0.60 (respectable)
• vs self_play_v1: > 0.70 (must have improved)

Step 4 — Report

Produce an Equilibria Evaluation Report containing:

1. Full cross-play win-rate matrix (versions × versions)
2. Cycle-forgetting verdict with historical checkpoint performance graph
3. Exploitability score with exploiter training curve
4. Population play win-rate table against all reference bots
5. Final Nash stability verdict: Stable / Marginal / Unstable

Additional Resources

Reference Files

• references/marl_equilibria_theory.md — Nash equilibrium formalism, exploitability metric derivation, cycle-forgetting literature

Scripts

• scripts/cross_play_matrix.py — Round-robin matchup runner with win-rate matrix generation
• scripts/exploitability_test.py — Exploiter training loop against frozen agent