spiral-zero-sum-game-reasoning

name	spiral-zero-sum-game-reasoning
title	SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn RL
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.24119
keywords	["ReinforcementLearning","SelfPlay","ZeroSumGames","ChainOfThought","Reasoning","LLM"]
description	A self-play framework enabling language models to develop sophisticated reasoning through competitive multi-turn games without human supervision. Achieves 10% improvement on reasoning benchmarks by training models to win against evolving opponents while maintaining interpretable thinking traces.

SPIRAL: Reasoning Through Competitive Self-Play on Zero-Sum Games

Language models trained with supervised learning struggle to develop genuinely sophisticated reasoning—they memorize patterns rather than learning to think through problems. Self-play on competitive games creates natural incentives for reasoning: a model must explain its strategy to convince itself (as the opponent) while also developing robust plans to win. SPIRAL demonstrates that this competitive framework outperforms supervised fine-tuning while requiring no human-annotated reasoning traces.

The core insight is that zero-sum games create a virtuous cycle: a stronger reasoning model becomes a tougher opponent, forcing continuous improvement. Unlike supervised learning where training data is fixed, self-play opponents dynamically evolve, preventing the model from exploiting dataset shortcuts and forcing genuine strategic thinking.

Core Concept

SPIRAL replaces supervised fine-tuning with a distributed self-play system where a single language model policy plays both sides of multiple two-player games, conditioned on player identity. The games are designed to require different reasoning skills—spatial reasoning, probabilistic inference, strategic negotiation—that transfer to academic benchmarks.

The key innovation is Role-Conditioned Advantage Estimation (RAE), which prevents "thinking collapse" where models abandon reasoning traces. RAE normalizes rewards relative to each player's expected performance, ensuring both players maintain interpretable chain-of-thought reasoning even as they improve.

Architecture Overview

• Distributed Actor-Learner System: Multiple actors generate trajectories through self-play; a central learner performs full-parameter updates with continuous opponent evolution
• Three Complementary Games: TicTacToe (spatial logic), Kuhn Poker (probabilistic reasoning), Simple Negotiation (strategic optimization)
• Role-Conditioned Advantage Estimation: Separate per-game, per-role baselines preventing reward variance from driving degenerate policies
• Multi-Game Training: Sequential or joint training across games with analysis of skill transfer
• Continuous Policy Updates: Both roles benefit from training improvements rather than fixed opponent strategies

Implementation

This implementation demonstrates the self-play training loop with role conditioning:

import torch
import torch.nn as nn
from typing import Tuple, 

 (nn.Module):
    
     ():
        ().__init__()
        .embeddings = nn.Embedding(vocab_size, hidden_dim)
        .role_embedding = nn.Embedding(, hidden_dim)  
        .transformer = TransformerCore(hidden_dim, depth=)
        .value_head = nn.Linear(hidden_dim, )

     ():
        
        
        state_embed = .embeddings(game_state_tokens)
        role_embed = .role_embedding(role)

        
        combined = state_embed + role_embed.unsqueeze()

        
        hidden = .transformer(combined)

        
        value = .value_head(hidden[:, -])
         hidden, value


 :
    
     ():
        
        .baselines = {}
        .baseline_optimizers = {}

         game_idx  (num_games):
             role  [, ]:
                key = (game_idx, role)
                baseline = nn.Linear(, )
                .baselines[key] = baseline
                .baseline_optimizers[key] = torch.optim.Adam(
                    baseline.parameters(), lr=learning_rate
                )

     ():
        
        key = (game_idx, role)
        baseline = .baselines[key]

        
        baseline_pred = baseline(hidden_states).squeeze(-)

        
        advantages = returns - baseline_pred.detach()

        
        baseline_loss = torch.mean((baseline_pred - returns) ** )
        .baseline_optimizers[key].zero_grad()
        baseline_loss.backward()
        .baseline_optimizers[key].step()

         advantages


 () -> [, , ]:
    
    trajectory = []
    game_state = initialize_game(game_type)
    current_role = 

     turn  (max_turns):
        
        state_tokens = encode_game_state(game_state)
        hidden, value = policy(state_tokens, role=current_role)

        
        action = sample_action(hidden)

        
        trajectory.append({
            : state_tokens,
            : current_role,
            : action,
            : value
        })

        
        game_state, reward, done = game_state.apply_action(action)

         done:
            

        
        current_role =  - current_role

    
    final_reward = game_state.get_reward_for_role()
    winner =   final_reward >   

     trajectory, final_reward, winner


 () -> :
    
    optimizer = torch.optim.Adam(policy.parameters(), lr=learning_rate)
    total_loss = 

     trajectory, final_reward, winner  trajectories:
        
        returns = []
        cumulative = 
         step  (((trajectory))):
            
             step == (trajectory) - :
                step_return = final_reward
            :
                step_return =  * cumulative
            returns.insert(, step_return)
            cumulative = step_return

        returns = torch.tensor(returns)

        
         step_idx, step_data  (trajectory):
            role = step_data[]
            hidden = step_data[].unsqueeze()

            
            advantage = rae.compute_advantage(
                returns[step_idx:step_idx+], game_idx, role, hidden
            )

            
            log_prob = compute_log_prob(step_data)
            loss = -log_prob * advantage.detach()
            total_loss += loss

    
    optimizer.zero_grad()
    total_loss.backward()
    optimizer.step()

     total_loss.item()