spice-corpus-self-play

name	spice-corpus-self-play
title	SPICE: Self-Play In Corpus Environments Improves Reasoning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.24684
keywords	["Self-play","Reasoning","Corpus Grounding","RL","Curriculum Learning"]
description	Enables continuous self-improvement through corpus-grounded self-play. Challenger mines difficult examples from document corpus for Reasoner to solve. External corpus prevents task stagnation that plagues closed-loop self-play. Achieves 8.9% math, 9.8% general reasoning improvements.

SPICE: Corpus-Grounded Self-Play for Reasoning

Self-play RL struggles with improvement plateaus because agents exhaust limited task spaces. SPICE grounds self-play in a real-world document corpus, enabling continuous challenge generation at the frontier of capability.

The corpus provides the rich, near-inexhaustible signal necessary for sustained reasoning improvement.

Core Concept

Two complementary roles:

• Challenger: mines difficult examples from corpus to create tasks
• Reasoner: solves Challenger-generated tasks

The Challenger creates automatic curriculum using corpus, addressing the fundamental limitation of closed-loop self-play: limited task diversity.

Architecture Overview

• Corpus of documents for task mining
• Challenger model: identifies difficult, solvable problems
• Reasoner model: attempts to solve mined problems
• Shared reward signal for co-training

Implementation Steps

Implement Challenger that mines tasks from corpus:

class ChallengerModel:
    def __init__(self, llm, corpus):
        self.llm = llm
        self.corpus = corpus

    def mine_task(self, reasoner_capability_level):
        """Generate task at boundary of reasoner capability."""
        # Sample documents from corpus
        candidates = self.corpus.sample(num_candidates=100)

        difficulty_scores = []
        for doc in candidates:
            # Extract problem from document
            problem = self.llm.extract_problem(doc)

            # Estimate difficulty relative to reasoner capability
            difficulty = self.estimate_difficulty(problem, reasoner_capability_level)

            difficulty_scores.append((problem, difficulty))

        # Select problems at frontier of capability
        
        target_difficulty = reasoner_capability_level + 
        frontier_problems = [
            p  p, d  difficulty_scores
             (d - target_difficulty) < 
        ]

         frontier_problems[]  frontier_problems  candidates[]

     ():
        
        
        problem_emb = .llm.embed(problem)

        
        similar_solved = []
         solved_problem  .reasoner_history:
            similarity = cosine_similarity(
                problem_emb,
                .llm.embed(solved_problem)
            )
            similar_solved.append(similarity)

        
        avg_similarity = (similar_solved) / (similar_solved)  similar_solved  
          - avg_similarity