| name | r-zero-self-evolving-reasoning |
| title | R-Zero - Self-Evolving Reasoning LLM from Zero Data |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2508.05004 |
| keywords | ["self-play","co-evolution","reasoning","data-generation","bootstrapping"] |
| description | Co-evolutionary framework where Challenger generates tasks and Solver solves them. Models evolve autonomously from scratch without human annotations. Improves math reasoning +6.49pts and general reasoning +7.54pts. |
R-Zero: Self-Evolving Reasoning LLM from Zero Data
Core Concept
R-Zero eliminates dependency on human-curated task datasets through autonomous co-evolution. Two models—a Challenger that generates progressively harder tasks and a Solver that tackles them—interact in a feedback loop that generates training data and improves both models without any initial labeled dataset. This enables models to break through baseline capabilities through self-generated, curriculum-learned tasks.
Architecture Overview
- Challenger Model: Generates tasks at the frontier of Solver capability
- Solver Model: Solves Challenger's tasks and learns from outcomes
- Co-evolutionary Feedback: Challenger rewarded for hard-but-solvable tasks; Solver for solving harder tasks
- Curriculum Learning: Task difficulty naturally increases as Solver improves
- Data Generation: Training data emerges from interaction, not annotation
Implementation Steps
Step 1: Initialize Challenger and Solver Models
Set up two models with appropriate initializations and objectives.
import torch
from typing import Dict, List, Tuple
class ChallengerModel(torch.nn.Module):
"""
Generates tasks at the frontier of solver capability.
"""
def __init__(self, base_model, task_domain="math"):
super().__init__()
self.base_model = base_model
self.task_domain = task_domain
self.task_buffer = []
self.difficulty_estimate = 0.5
def generate_task(self, difficulty_target: float) -> str:
"""
Generate a task at target difficulty.
Args:
difficulty_target: Target difficulty 0-1 (0=trivial, 1=impossible)
Returns:
Task prompt
"""
prompt = f"""
Generate a {self.task_domain} problem.
Difficulty target: {difficulty_target:.1f}/1.0
- 0.0: Trivial, easily solvable
- 0.5: Medium difficulty, requires some reasoning
- 1.0: Very hard, near impossible
Ensure the problem is clear and has a verifiable solution.
Problem:
"""
task = self.base_model.generate(prompt, max_length=300)
return task
def estimate_task_difficulty(self, task: str, solver_accuracy: float) -> float:
"""
Estimate task difficulty based on solver performance.
Args:
task: Task description
solver_accuracy: Proportion of attempts solver succeeds
Returns:
Estimated difficulty 0-1
"""
estimated_difficulty = - solver_accuracy
estimated_difficulty
() -> :
target_success_rate =
difficulty = - solver_accuracy
reward = -(difficulty - target_success_rate) **
reward
Step 2: Implement Solver Model
Create a model that solves Challenger tasks and provides learning signal.
class SolverModel(torch.nn.Module):
"""
Solves Challenger's tasks and learns from outcomes.
"""
def __init__(self, base_model, task_domain="math"):
super().__init__()
self.base_model = base_model
self.task_domain = task_domain
self.solution_buffer = []
self.performance_history = []
def solve_task(self, task: str, max_attempts: int = 1) -> Tuple[str, bool]:
"""
Attempt to solve a task.
Args:
task: Task prompt
max_attempts: Number of solution attempts
Returns:
(solution, is_correct)
"""
prompt = f"""
Solve this {self.task_domain} problem:
{task}
Show your work and provide final answer.
Solution:
"""
solution = self.base_model.generate(prompt, max_length=500)
is_correct = self.verify_solution(task, solution)
return solution, is_correct
def verify_solution(self, task: str, solution: str) -> bool:
"""
Verify correctness of solution.
Args:
task: Original task
solution: Proposed solution
Returns:
True if solution is correct
"""
verification_prompt =
response = .base_model.generate(verification_prompt, max_length=)
response.upper()
() -> :
is_correct:
task_difficulty = .estimate_task_difficulty(task)
reward = task_difficulty
:
reward = -
reward
() -> :
complexity_score = ((task.split()), ) /
complexity_score
():
i (, (tasks), batch_size):
batch_tasks = tasks[i:i+batch_size]
task batch_tasks:
solution, is_correct = .solve_task(task)
is_correct:
.train_on_solution(task, solution)
():
prompt =
.base_model.finetune(prompt, solution)
Step 3: Implement Co-evolutionary Training Loop
Create the main loop where Challenger and Solver interact.
class CoevolutionaryFramework:
"""
Framework orchestrating Challenger-Solver co-evolution.
"""
def __init__(self, challenger: ChallengerModel, solver: SolverModel):
self.challenger = challenger
self.solver = solver
self.generation = 0
self.task_archive = []
self.performance_log = []
def run_generation(self, num_tasks: int = 20) -> Dict:
"""
Run one generation of co-evolution.
Args:
num_tasks: Number of tasks to generate and solve
Returns:
Metrics for this generation
"""
self.generation += 1
tasks_generated = []
tasks_solved_correctly = 0
challenger_rewards = []
solver_rewards = []
base_difficulty = self._compute_difficulty_target()
for task_idx in range(num_tasks):
task = self.challenger.generate_task(difficulty_target=base_difficulty)
tasks_generated.append(task)
solution, is_correct = self.solver.solve_task(task)
if is_correct:
tasks_solved_correctly += 1
solver_success_rate = tasks_solved_correctly / (task_idx + 1)
challenger_reward = .challenger.compute_reward(
task,
is_correct,
solver_success_rate
)
challenger_rewards.append(challenger_reward)
solver_reward = .solver.compute_reward(task, solution, is_correct)
solver_rewards.append(solver_reward)
.task_archive.append({
: task,
: solution,
: is_correct,
: .generation
})
correct_tasks = [t[] t .task_archive
t[] t[] == .generation]
.solver.learn_from_tasks(correct_tasks)
._update_models_from_rewards(challenger_rewards, solver_rewards)
metrics = {
: .generation,
: (tasks_generated),
: tasks_solved_correctly,
: tasks_solved_correctly / num_tasks,
: (challenger_rewards) / (challenger_rewards),
: (solver_rewards) / (solver_rewards),
: base_difficulty
}
.performance_log.append(metrics)
metrics
() -> :
.performance_log:
recent_success = .performance_log[-][]
recent_success > :
new_difficulty = (, .performance_log[-][] * )
recent_success < :
new_difficulty = (, .performance_log[-][] / )
:
new_difficulty = .performance_log[-][]
new_difficulty
():
():
gen (num_generations):
metrics = .run_generation(num_tasks=)
(
)
.performance_log
Step 4: Evaluate Final Performance
Benchmark the evolved model on standard benchmarks.
def evaluate_r_zero_model(solver_model, benchmark_suites: Dict[str, List]) -> Dict:
"""
Evaluate R-Zero trained model on benchmarks.
Args:
solver_model: Evolved solver model
benchmark_suites: Different benchmark suites (math, reasoning, etc.)
Returns:
Performance on each benchmark
"""
results = {}
for benchmark_name, problems in benchmark_suites.items():
correct = 0
for problem in problems:
solution, is_correct = solver_model.solve_task(problem["prompt"])
if is_correct:
correct += 1
accuracy = correct / len(problems)
results[benchmark_name] = accuracy
print(f"{benchmark_name}: {accuracy:.1%}")
return results
Practical Guidance
When to Use R-Zero
- No human annotations available: Bootstrapping models from scratch
- Continuous learning scenarios: Models that improve by self-play
- Curriculum learning applications: Progressive task difficulty needed
- Research on self-improvement: Understanding autonomous capability growth
When NOT to Use R-Zero
- Abundant labeled data: Supervised learning likely more efficient
- Fixed task distribution: No need for curriculum generation
- Real-time constraints: Co-evolution is slow and compute-intensive
- Safety-critical systems: Hard to control generated task distribution
Hyperparameter Recommendations
- Target solver success rate: 0.3-0.5 (challenging but solvable)
- Difficulty increase factor: 1.05-1.15 per generation
- Task batch size: 16-32 tasks per generation
- Number of generations: 10-20 for meaningful improvement
- Tasks per generation: 20-50 depends on task complexity
Key Insights
The critical insight is that task generation and solving can bootstrap each other. By rewarding the Challenger for hard-but-solvable tasks, the framework naturally creates a curriculum. The Solver improves by learning from successfully solved tasks, which in turn motivates the Challenger to generate harder problems. This virtuous cycle enables capability growth without human annotation.
Reference
R-Zero: Self-Evolving Reasoning LLM from Zero Data (arXiv:2508.05004)
Introduces co-evolutionary framework with Challenger and Solver models that generate training data and improve autonomously. Demonstrates +6.49pts math and +7.54pts general reasoning improvement without pre-existing labeled data.
