rlad-abstract-discovery-reasoning

name	rlad-abstract-discovery-reasoning
title	RLAD: Learning to Discover Abstractions via Reasoning RL
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.02263
keywords	["reasoning-abstraction","RLVR","strategy-discovery","curriculum-learning"]
description	Train reasoning models to discover diverse solution strategies through two-player RL that jointly optimizes abstraction generation and solution derivation. Use when exploring multiple reasoning approaches is preferable to single-strategy depth.

RLAD: Learning to Discover Abstractions via Reasoning RL

RLAD introduces a two-player RL framework where abstraction generators and solution generators co-evolve. This addresses a fundamental inefficiency: RL often optimizes for solution depth rather than strategy diversity. By explicitly rewarding abstraction diversity, models explore broader solution landscapes.

Core Architecture

• Reasoning abstractions: Concise natural language descriptions of procedural/factual knowledge
• Two-player dynamics: Generator vs. Verifier with competing objectives
• Abstraction-guided generation: Solutions generated conditioned on discovered abstractions
• Modified reward system: Prevents failure modes (e.g., gaming with trivial abstractions)
• Multi-domain generalization: Works across math, coding, and diverse reasoning domains

Implementation Steps

Setup two-player RL framework for abstraction discovery:

# Initialize RLAD trainer with abstraction and solution components
from rlad import AbstractionRL, SolutionRL, TwoPlayerOptimizer

# Create abstraction generator
abstraction_generator = AbstractionRL(
    model="gpt-4-mini",  # or stronger model for warm-starting
    max_abstraction_length=256,
    abstraction_style="knowledge_based"
)

# Create solution generator conditioned on abstractions
solution_generator = SolutionRL(
    model="your_reasoning_llm",
    conditioning="abstraction_aware",
    max_solution_steps=50
)

# Setup two-player optimizer
optimizer = TwoPlayerOptimizer(
    abstraction_generator=abstraction_generator,
    solution_generator=solution_generator,
    reward_mode="abstraction_diversity"
)

Execute two-player RL training:

# Training loop optimizing both abstractors and solvers
for epoch in range(num_epochs):
    for batch in training_data:
        problem = batch[]
        ground_truth = batch[]

        
        abstractions = abstraction_generator.sample(
            problem=problem,
            num_samples=,
            temperature=,  
            beam_size=    
        )

        
        solutions = []
        rewards = []

         abstraction  abstractions:
            solution = solution_generator.generate(
                problem=problem,
                abstraction=abstraction,
                max_length=,
                temperature=
            )
            solutions.append(solution)

            
            is_correct = verify_solution(solution, ground_truth)
            reward =   is_correct  
            rewards.append(reward)

        
        abstraction_reward = compute_diversity_score(abstractions)

        
        
         i, abstraction  (abstractions):
             rewards[i] == :
                abstraction_reward *=   

        
        abstraction_loss = optimizer.compute_abstraction_loss(
            abstractions=abstractions,
            diversity_reward=abstraction_reward,
            solution_success=rewards
        )

        solution_loss = optimizer.compute_solution_loss(
            solutions=solutions,
            targets=ground_truth,
            abstractions=abstractions
        )

        
        (abstraction_loss + solution_loss).backward()
        optimizer.step()