reasoning-gym-verifiable-rewards

name	reasoning-gym-verifiable-rewards
title	REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2505.24760
keywords	["Reinforcement Learning","Verifiable Rewards","Reasoning","Procedural Generation"]
description	Create infinite training environments for reasoning with automatic verification using procedural generation and domain-specific evaluators.

Reasoning Gym: Train RL on Infinite Verifiable Reasoning Tasks

Standard RL for reasoning bottlenecks on fixed datasets: collect 10k problems, train until convergence, hit a ceiling. Reasoning Gym inverts this by providing procedurally generated reasoning environments with automatic correctness verification. Generate virtually infinite algebra problems, logic puzzles, geometry proofs, and games at tunable difficulty. Each problem includes a verifier that checks solutions automatically, providing reliable reward signals for RL. Train continuously on increasingly difficult problems without ever repeating an example.

This enables continuous curriculum learning where models improve across escalating complexity, and eliminates the data collection bottleneck that limits reasoning research.

Core Concept

Procedural generation + automatic verification = infinite training signal. For each domain (algebra, logic, geometry, etc.), implement a generator that produces valid problem instances with adjustable parameters, and a verifier that evaluates solution correctness deterministically. This decouples problem quantity from manual annotation effort. As models improve, increase difficulty parameters; environments adapt to learner progress automatically.

Architecture Overview

• Procedural Generators: Per-domain problem generators (algebra equations, logic formulas, geometry diagrams) parameterized by difficulty
• Verifiers: Deterministic correctness checkers for each domain (symbolic equation solvers, proof validators, game rule checkers)
• Difficulty Controller: Automatically adjusts problem complexity based on learner performance (curriculum learning)
• Multi-Domain Coverage: 100+ generators spanning math, logic, games, cognition (sorting, counting), and more
• RL Integration: Seamless integration with standard RL algorithms (PPO, GRPO) providing reward = correctness signal

Implementation

This implementation demonstrates procedural generation with verification for key reasoning domains.

Implement algebra problem generation and verification:

import random
import sympy as sp
from typing import Tuple, Dict
from dataclasses import dataclass
from abc import ABC, abstractmethod

@dataclass
class Problem:
    task_id: str
    problem_text: str
    solution_correct: bool
    difficulty: float

class ReasoningEnvironment(ABC):
    """Base class for procedurally generated reasoning environments."""

    @abstractmethod
    def generate_problem(self, difficulty: float) -> Problem:
        """Generate a problem at given difficulty level."""
        pass

    @abstractmethod
    def verify_solution(self, problem: Problem, solution: str) -> bool:
        """Verify if solution is correct."""
        pass

class AlgebraEnvironment(ReasoningEnvironment):
    """Linear and quadratic equation solving."""

    def __init__(self, seed: int = 42):
        random.seed(seed)

    def generate_problem() -> Problem:
        
         difficulty < :
            
            a = random.randint(, )
            b = random.randint(-, )
            c = random.randint(-, )
            problem_text = 
            solution_value = (c - b) / a

         difficulty < :
            
            a = random.randint(, )
            b = random.randint(, )
            c = random.randint(-, )
            problem_text = 
            solution_value = c / (a + b)

        :
            
            a = random.randint(, )
            b = random.randint(-, )
            c = random.randint(-, )
            problem_text = 
            
            x = sp.Symbol()
            eq = a*x** + b*x + c
            solutions = sp.solve(eq, x)
            solution_value = (solutions[])  solutions  

         Problem(
            task_id=,
            problem_text=problem_text,
            solution_correct=,
            difficulty=difficulty
        )

     () -> :
        
        :
            
                solution_str:
                 

            answer_part = solution_str.split()[-].strip()
            proposed_answer = (answer_part)

            
            problem_text = problem.problem_text
                problem_text:
                 

            eq_str = problem_text.split()[-].strip()
            x = sp.Symbol()

            
            eq = sp.sympify(eq_str)
            solutions = sp.solve(eq, x)

            
             sol  solutions:
                 ((sol) - proposed_answer) < :
                     
             

         (ValueError, SyntaxError, sp.SympifyError):
             

 ():
    

     ():
        random.seed(seed)
        .predicates = [, , , ]
        .entities = [, , , ]

     () -> Problem:
        
         difficulty < :
            
            propositions = [, , ]
            selected = random.sample(propositions, )
            op = random.choice([, , ])
            problem_text = 

        :
            
            pred = random.choice(.predicates)
            e1, e2 = random.sample(.entities, )
            problem_text = 

         Problem(
            task_id=,
            problem_text=problem_text,
            solution_correct=,
            difficulty=difficulty
        )

     () -> :
        
        :
            
            answer = solution_str.strip().upper()
               answer  answer == :
                   
               answer  answer == :
                 
             

         Exception:
             


algebra_env = AlgebraEnvironment()
logic_env = LogicEnvironment()


 difficulty  [, , ]:
    alg_prob = algebra_env.generate_problem(difficulty)
    ()

    logic_prob = logic_env.generate_problem(difficulty)
    ()
    ()

Build a curriculum learning controller that adapts difficulty:

class CurriculumController:
    """Automatically adjust problem difficulty based on performance."""

    def __init__(self, initial_difficulty: float = 0.2,
                 performance_window: int = 100):
        self.current_difficulty = initial_difficulty
        self.performance_window = performance_window
        self.recent_performance = []
        self.episode_count = 0

    def update_performance(self, was_correct: bool):
        """Record episode result."""
        self.recent_performance.append(was_correct)
        if len(self.recent_performance) > self.performance_window:
            self.recent_performance.pop(0)
        self.episode_count += 1

    def get_current_difficulty(self) -> float:
        """Return current difficulty level (0-1)."""
        return min(1.0, self.current_difficulty)

    def adjust_difficulty(self):
        """Increase/decrease difficulty based on recent performance."""
        if len(.recent_performance) < .performance_window:
              

        recent_accuracy = (.recent_performance) / (.recent_performance)

        
         recent_accuracy > :
            
            .current_difficulty = (, .current_difficulty + )
         recent_accuracy < :
            
            .current_difficulty = (, .current_difficulty - )

     ():
        
        .update_performance(was_correct)
         .episode_count %  == :
            .adjust_difficulty()


curriculum = CurriculumController(initial_difficulty=)

 episode  ():
    
    difficulty = curriculum.get_current_difficulty()
    problem = algebra_env.generate_problem(difficulty)

    
    agent_solution =   

    
    is_correct = algebra_env.verify_solution(problem, agent_solution)

    
    curriculum.episode_done(is_correct)

     (episode + ) %  == :
        (
              )

Build a multi-domain reasoning gym:

class ReasoningGym:
    """Multi-domain reasoning environment with curriculum learning."""

    def __init__(self):
        self.environments = {
            "algebra": AlgebraEnvironment(),
            "logic": LogicEnvironment(),
            # Add more domains: geometry, arithmetic, games, etc.
        }
        self.curriculums = {
            domain: CurriculumController()
            for domain in self.environments
        }
        self.current_domain = None

    def sample_task(self, domain: str = None) -> Tuple[Problem, str]:
        """Sample a reasoning task at curriculum-appropriate difficulty."""
        if domain is None:
            # Sample domain uniformly
            domain = random.choice(list(self.environments.keys()))

        self.current_domain = domain
        env = self.environments[domain]
        curriculum = self.curriculums[domain]

        difficulty = curriculum.get_current_difficulty()
        problem = env.generate_problem(difficulty)

        return problem, domain

    def evaluate_solution(self, solution: str) -> bool:
        """Verify solution for current task."""
        env = .environments[.current_domain]
         env.verify_solution(problem, solution)

     () -> [, ]:
        
        is_correct = .evaluate_solution(solution)
        reward =   is_correct  

        curriculum = .curriculums[.current_domain]
        curriculum.episode_done(is_correct)

         reward,   


gym = ReasoningGym()


 step  ():
    problem, domain = gym.sample_task()

    
     random.random() < :
        agent_solution =   
    :
        agent_solution =   

    reward, done = gym.step(agent_solution)

     (step + ) %  == :
        ()
         domain, curr  gym.curriculums.items():
            ()

Practical Guidance

Aspect	Details
Initial Difficulty	Start at 0.1-0.3; too hard causes early failure, too easy wastes steps
Performance Window	50-200 episodes; balance responsiveness to improvement vs. noise
Difficulty Step Size	0.05 per adjustment; 0.02 for fine-grained, 0.10 for coarse
Domain Selection	Sample uniformly or weight by current performance (easier domains = boost signal)
Verifier Reliability	Test verifiers against ground truth before training; bugs here invalidate entire RL signal

When to Use:

• RL on reasoning tasks where you want continuous improvement without data collection ceiling
• Research comparing RL algorithms: eliminate dataset variance by using same procedural env
• Curriculum learning: automatically escalate problem difficulty as model improves
• Multi-task learning: train single model across domains with automatic mixing
• Scaling RL training indefinitely without hitting dataset limits

When NOT to Use:

• Domain has no natural parametric difficulty (open-ended creative tasks)
• Verifier is expensive to run (slows down RL training significantly)
• Need to evaluate on real-world distribution: procedural env may not match target
• Domains with sparse ground truth (open-ended generation, dialogue)

Common Pitfalls:

• Verifier bugs: wrong reward signals corrupt entire training; test extensively first
• Difficulty oscillation: curriculum adjusts too aggressively; use larger performance windows
• Domain imbalance: some procedural generators produce easier/harder problems; normalize difficulty
• Overfitting to procedural structure: models learn to exploit generator quirks; randomize problem generation thoroughly

Reference

REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards https://arxiv.org/abs/2505.24760