| name | spiral-zero-sum-game-reasoning |
| title | SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn RL |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2506.24119 |
| keywords | ["ReinforcementLearning","SelfPlay","ZeroSumGames","ChainOfThought","Reasoning","LLM"] |
| description | A self-play framework enabling language models to develop sophisticated reasoning through competitive multi-turn games without human supervision. Achieves 10% improvement on reasoning benchmarks by training models to win against evolving opponents while maintaining interpretable thinking traces. |
SPIRAL: Reasoning Through Competitive Self-Play on Zero-Sum Games
Language models trained with supervised learning struggle to develop genuinely sophisticated reasoning—they memorize patterns rather than learning to think through problems. Self-play on competitive games creates natural incentives for reasoning: a model must explain its strategy to convince itself (as the opponent) while also developing robust plans to win. SPIRAL demonstrates that this competitive framework outperforms supervised fine-tuning while requiring no human-annotated reasoning traces.
The core insight is that zero-sum games create a virtuous cycle: a stronger reasoning model becomes a tougher opponent, forcing continuous improvement. Unlike supervised learning where training data is fixed, self-play opponents dynamically evolve, preventing the model from exploiting dataset shortcuts and forcing genuine strategic thinking.
Core Concept
SPIRAL replaces supervised fine-tuning with a distributed self-play system where a single language model policy plays both sides of multiple two-player games, conditioned on player identity. The games are designed to require different reasoning skills—spatial reasoning, probabilistic inference, strategic negotiation—that transfer to academic benchmarks.
The key innovation is Role-Conditioned Advantage Estimation (RAE), which prevents "thinking collapse" where models abandon reasoning traces. RAE normalizes rewards relative to each player's expected performance, ensuring both players maintain interpretable chain-of-thought reasoning even as they improve.
Architecture Overview
- Distributed Actor-Learner System: Multiple actors generate trajectories through self-play; a central learner performs full-parameter updates with continuous opponent evolution
- Three Complementary Games: TicTacToe (spatial logic), Kuhn Poker (probabilistic reasoning), Simple Negotiation (strategic optimization)
- Role-Conditioned Advantage Estimation: Separate per-game, per-role baselines preventing reward variance from driving degenerate policies
- Multi-Game Training: Sequential or joint training across games with analysis of skill transfer
- Continuous Policy Updates: Both roles benefit from training improvements rather than fixed opponent strategies
Implementation
This implementation demonstrates the self-play training loop with role conditioning:
import torch
import torch.nn as nn
from typing import Tuple,
(nn.Module):
():
().__init__()
.embeddings = nn.Embedding(vocab_size, hidden_dim)
.role_embedding = nn.Embedding(, hidden_dim)
.transformer = TransformerCore(hidden_dim, depth=)
.value_head = nn.Linear(hidden_dim, )
():
state_embed = .embeddings(game_state_tokens)
role_embed = .role_embedding(role)
combined = state_embed + role_embed.unsqueeze()
hidden = .transformer(combined)
value = .value_head(hidden[:, -])
hidden, value
:
():
.baselines = {}
.baseline_optimizers = {}
game_idx (num_games):
role [, ]:
key = (game_idx, role)
baseline = nn.Linear(, )
.baselines[key] = baseline
.baseline_optimizers[key] = torch.optim.Adam(
baseline.parameters(), lr=learning_rate
)
():
key = (game_idx, role)
baseline = .baselines[key]
baseline_pred = baseline(hidden_states).squeeze(-)
advantages = returns - baseline_pred.detach()
baseline_loss = torch.mean((baseline_pred - returns) ** )
.baseline_optimizers[key].zero_grad()
baseline_loss.backward()
.baseline_optimizers[key].step()
advantages
() -> [, , ]:
trajectory = []
game_state = initialize_game(game_type)
current_role =
turn (max_turns):
state_tokens = encode_game_state(game_state)
hidden, value = policy(state_tokens, role=current_role)
action = sample_action(hidden)
trajectory.append({
: state_tokens,
: current_role,
: action,
: value
})
game_state, reward, done = game_state.apply_action(action)
done:
current_role = - current_role
final_reward = game_state.get_reward_for_role()
winner = final_reward >
trajectory, final_reward, winner
() -> :
optimizer = torch.optim.Adam(policy.parameters(), lr=learning_rate)
total_loss =
trajectory, final_reward, winner trajectories:
returns = []
cumulative =
step (((trajectory))):
step == (trajectory) - :
step_return = final_reward
:
step_return = * cumulative
returns.insert(, step_return)
cumulative = step_return
returns = torch.tensor(returns)
step_idx, step_data (trajectory):
role = step_data[]
hidden = step_data[].unsqueeze()
advantage = rae.compute_advantage(
returns[step_idx:step_idx+], game_idx, role, hidden
)
log_prob = compute_log_prob(step_data)
loss = -log_prob * advantage.detach()
total_loss += loss
optimizer.zero_grad()
total_loss.backward()
optimizer.step()
total_loss.item()
