| name | reasoning-gym-verifiable-rewards |
| title | REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2505.24760 |
| keywords | ["Reinforcement Learning","Verifiable Rewards","Reasoning","Procedural Generation"] |
| description | Create infinite training environments for reasoning with automatic verification using procedural generation and domain-specific evaluators. |
Reasoning Gym: Train RL on Infinite Verifiable Reasoning Tasks
Standard RL for reasoning bottlenecks on fixed datasets: collect 10k problems, train until convergence, hit a ceiling. Reasoning Gym inverts this by providing procedurally generated reasoning environments with automatic correctness verification. Generate virtually infinite algebra problems, logic puzzles, geometry proofs, and games at tunable difficulty. Each problem includes a verifier that checks solutions automatically, providing reliable reward signals for RL. Train continuously on increasingly difficult problems without ever repeating an example.
This enables continuous curriculum learning where models improve across escalating complexity, and eliminates the data collection bottleneck that limits reasoning research.
Core Concept
Procedural generation + automatic verification = infinite training signal. For each domain (algebra, logic, geometry, etc.), implement a generator that produces valid problem instances with adjustable parameters, and a verifier that evaluates solution correctness deterministically. This decouples problem quantity from manual annotation effort. As models improve, increase difficulty parameters; environments adapt to learner progress automatically.
Architecture Overview
- Procedural Generators: Per-domain problem generators (algebra equations, logic formulas, geometry diagrams) parameterized by difficulty
- Verifiers: Deterministic correctness checkers for each domain (symbolic equation solvers, proof validators, game rule checkers)
- Difficulty Controller: Automatically adjusts problem complexity based on learner performance (curriculum learning)
- Multi-Domain Coverage: 100+ generators spanning math, logic, games, cognition (sorting, counting), and more
- RL Integration: Seamless integration with standard RL algorithms (PPO, GRPO) providing reward = correctness signal
Implementation
This implementation demonstrates procedural generation with verification for key reasoning domains.
Implement algebra problem generation and verification:
import random
import sympy as sp
from typing import Tuple, Dict
from dataclasses import dataclass
from abc import ABC, abstractmethod
@dataclass
class Problem:
task_id: str
problem_text: str
solution_correct: bool
difficulty: float
class ReasoningEnvironment(ABC):
"""Base class for procedurally generated reasoning environments."""
@abstractmethod
def generate_problem(self, difficulty: float) -> Problem:
"""Generate a problem at given difficulty level."""
pass
@abstractmethod
def verify_solution(self, problem: Problem, solution: str) -> bool:
"""Verify if solution is correct."""
pass
class AlgebraEnvironment(ReasoningEnvironment):
"""Linear and quadratic equation solving."""
def __init__(self, seed: int = 42):
random.seed(seed)
def generate_problem() -> Problem:
difficulty < :
a = random.randint(, )
b = random.randint(-, )
c = random.randint(-, )
problem_text =
solution_value = (c - b) / a
difficulty < :
a = random.randint(, )
b = random.randint(, )
c = random.randint(-, )
problem_text =
solution_value = c / (a + b)
:
a = random.randint(, )
b = random.randint(-, )
c = random.randint(-, )
problem_text =
x = sp.Symbol()
eq = a*x** + b*x + c
solutions = sp.solve(eq, x)
solution_value = (solutions[]) solutions
Problem(
task_id=,
problem_text=problem_text,
solution_correct=,
difficulty=difficulty
)
() -> :
:
solution_str:
answer_part = solution_str.split()[-].strip()
proposed_answer = (answer_part)
problem_text = problem.problem_text
problem_text:
eq_str = problem_text.split()[-].strip()
x = sp.Symbol()
eq = sp.sympify(eq_str)
solutions = sp.solve(eq, x)
sol solutions:
((sol) - proposed_answer) < :
(ValueError, SyntaxError, sp.SympifyError):
():
():
random.seed(seed)
.predicates = [, , , ]
.entities = [, , , ]
() -> Problem:
difficulty < :
propositions = [, , ]
selected = random.sample(propositions, )
op = random.choice([, , ])
problem_text =
:
pred = random.choice(.predicates)
e1, e2 = random.sample(.entities, )
problem_text =
Problem(
task_id=,
problem_text=problem_text,
solution_correct=,
difficulty=difficulty
)
() -> :
:
answer = solution_str.strip().upper()
answer answer == :
answer answer == :
Exception:
algebra_env = AlgebraEnvironment()
logic_env = LogicEnvironment()
difficulty [, , ]:
alg_prob = algebra_env.generate_problem(difficulty)
()
logic_prob = logic_env.generate_problem(difficulty)
()
()
Build a curriculum learning controller that adapts difficulty:
class CurriculumController:
"""Automatically adjust problem difficulty based on performance."""
def __init__(self, initial_difficulty: float = 0.2,
performance_window: int = 100):
self.current_difficulty = initial_difficulty
self.performance_window = performance_window
self.recent_performance = []
self.episode_count = 0
def update_performance(self, was_correct: bool):
"""Record episode result."""
self.recent_performance.append(was_correct)
if len(self.recent_performance) > self.performance_window:
self.recent_performance.pop(0)
self.episode_count += 1
def get_current_difficulty(self) -> float:
"""Return current difficulty level (0-1)."""
return min(1.0, self.current_difficulty)
def adjust_difficulty(self):
"""Increase/decrease difficulty based on recent performance."""
if len(.recent_performance) < .performance_window:
recent_accuracy = (.recent_performance) / (.recent_performance)
recent_accuracy > :
.current_difficulty = (, .current_difficulty + )
recent_accuracy < :
.current_difficulty = (, .current_difficulty - )
():
.update_performance(was_correct)
.episode_count % == :
.adjust_difficulty()
curriculum = CurriculumController(initial_difficulty=)
episode ():
difficulty = curriculum.get_current_difficulty()
problem = algebra_env.generate_problem(difficulty)
agent_solution =
is_correct = algebra_env.verify_solution(problem, agent_solution)
curriculum.episode_done(is_correct)
(episode + ) % == :
(
)
Build a multi-domain reasoning gym:
class ReasoningGym:
"""Multi-domain reasoning environment with curriculum learning."""
def __init__(self):
self.environments = {
"algebra": AlgebraEnvironment(),
"logic": LogicEnvironment(),
}
self.curriculums = {
domain: CurriculumController()
for domain in self.environments
}
self.current_domain = None
def sample_task(self, domain: str = None) -> Tuple[Problem, str]:
"""Sample a reasoning task at curriculum-appropriate difficulty."""
if domain is None:
domain = random.choice(list(self.environments.keys()))
self.current_domain = domain
env = self.environments[domain]
curriculum = self.curriculums[domain]
difficulty = curriculum.get_current_difficulty()
problem = env.generate_problem(difficulty)
return problem, domain
def evaluate_solution(self, solution: str) -> bool:
"""Verify solution for current task."""
env = .environments[.current_domain]
env.verify_solution(problem, solution)
() -> [, ]:
is_correct = .evaluate_solution(solution)
reward = is_correct
curriculum = .curriculums[.current_domain]
curriculum.episode_done(is_correct)
reward,
gym = ReasoningGym()
step ():
problem, domain = gym.sample_task()
random.random() < :
agent_solution =
:
agent_solution =
reward, done = gym.step(agent_solution)
(step + ) % == :
()
domain, curr gym.curriculums.items():
()
Practical Guidance
| Aspect | Details |
|---|
| Initial Difficulty | Start at 0.1-0.3; too hard causes early failure, too easy wastes steps |
| Performance Window | 50-200 episodes; balance responsiveness to improvement vs. noise |
| Difficulty Step Size | 0.05 per adjustment; 0.02 for fine-grained, 0.10 for coarse |
| Domain Selection | Sample uniformly or weight by current performance (easier domains = boost signal) |
| Verifier Reliability | Test verifiers against ground truth before training; bugs here invalidate entire RL signal |
When to Use:
- RL on reasoning tasks where you want continuous improvement without data collection ceiling
- Research comparing RL algorithms: eliminate dataset variance by using same procedural env
- Curriculum learning: automatically escalate problem difficulty as model improves
- Multi-task learning: train single model across domains with automatic mixing
- Scaling RL training indefinitely without hitting dataset limits
When NOT to Use:
- Domain has no natural parametric difficulty (open-ended creative tasks)
- Verifier is expensive to run (slows down RL training significantly)
- Need to evaluate on real-world distribution: procedural env may not match target
- Domains with sparse ground truth (open-ended generation, dialogue)
Common Pitfalls:
- Verifier bugs: wrong reward signals corrupt entire training; test extensively first
- Difficulty oscillation: curriculum adjusts too aggressively; use larger performance windows
- Domain imbalance: some procedural generators produce easier/harder problems; normalize difficulty
- Overfitting to procedural structure: models learn to exploit generator quirks; randomize problem generation thoroughly
Reference
REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards
https://arxiv.org/abs/2505.24760
