Train LLMs to discover domain-specific abstractions (concise representations of procedural and factual knowledge) through two-player RL. An abstraction generator proposes key insights, a solution generator uses them to solve problems, and both receive RL rewards, learning structured reasoning that transfers to harder problems.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
Train LLMs to discover domain-specific abstractions (concise representations of procedural and factual knowledge) through two-player RL. An abstraction generator proposes key insights, a solution generator uses them to solve problems, and both receive RL rewards, learning structured reasoning that transfers to harder problems.
RLAD: Two-Player RL for Abstraction Discovery
Long reasoning traces often become verbose and circular. A 50-token chain-of-thought might say the same thing three times before reaching an answer. The issue is that raw step-by-step reasoning doesn't isolate what's actually useful—the key insights or procedural knowledge that solve the problem.
RLAD approaches this differently. Instead of generating long traces, it trains two players: one discovers abstractions (concise summaries of useful knowledge), and one uses those abstractions to solve problems. Both learn through RL, creating a natural curriculum where abstractions must be genuinely useful to receive reward.
Core Concept
RLAD is a two-player system:
Abstraction Generator: Proposes 1-3 concise descriptions of key insights (e.g., "Fibonacci numbers grow exponentially," "Even numbers are divisible by 2")
Solution Generator: Receives abstractions as context, uses them to solve the problem
Reward signal: Solution generator gets reward for correctness; abstraction generator gets reward when its abstractions help solution generation
The key insight: abstractions must be useful. An abstraction only receives reward if it actually helps solve the problem, not for being verbose or clever.
Architecture Overview
Abstraction generator: Encoder-decoder or autoregressive model generating concise descriptions
Solution generator: Uses proposed abstractions as part of reasoning context
Verifier: Checks if solution is correct
Reward aggregator: Assigns credit to abstractions based on solution success
Training loop: Two-player RL with shared rewards
Implementation Steps
First, implement the abstraction generation component:
"""
Generate multiple abstractions for a problem.
Args:
problem: Problem statement (text)
num_abstractions: How many abstractions to generate
Returns:
abstractions: List of text descriptions
"""
f"""Given this problem, what are the key insights or knowledge needed to solve it?
Problem: {problem}
Generate {num_abstractions} concise abstractions (1 sentence each) that capture important insights.
Format:
[ABSTRACTION 1]: <description>
[ABSTRACTION 2]: <description>
etc.
"""
for
in
range
# Generate one abstraction at a time for diversity
self
self
# Extract abstraction from output
'\n'
for
in
if
'[ABSTRACTION'
in
and
':'
in
':'
1
1
break
return
def
score_abstraction_usefulness
self, abstraction, problem, solution_success
"""
Simple heuristic: abstraction is useful if it's mentioned in correct solution.
(In practice, would use more sophisticated metrics.)
Args:
abstraction: Generated abstraction text
problem: Problem statement
solution_success: Did the solution succeed?
Returns:
usefulness_score: 0-1 scalar
"""
# This is a placeholder; real scoring would be more sophisticated
# (e.g., measuring if solution leverages the abstraction)
if
# Reward abstractions that are concise and conceptually sound
min
1.0
30.0
len
# Shorter is better
return
else
return
0.0
Now implement the solution generator that uses abstractions:
classSolutionGenerator:
"""
Generate solutions given problem and abstractions.
"""def__init__(self, model, verifier):
self.model = model
self.verifier = verifier
defsolve_with_abstractions(self, problem, abstractions):
"""
Generate solution using provided abstractions as context.
Args:
problem: Problem statement
abstractions: List of key abstractions
Returns:
solution: Generated solution
reasoning: Full reasoning trace
"""# Build prompt that includes abstractions
abstraction_text = "\n".join(
[f"- {abs}"forabsin abstractions]
)
prompt = f"""Use these key insights to solve the problem:
{abstraction_text}
Problem: {problem}
Solve step by step:"""# Generate solution
output = self.model.generate(prompt, max_length=200)
return output
defscore_solution(self, solution, problem):
"""
Check if solution is correct.
Args:
solution: Proposed solution
problem: Original problem
Returns:
is_correct: Binary correctness
confidence: 0-1 confidence score
"""
is_correct = self.verifier.verify(solution, problem)
confidence = self.verifier.get_confidence(solution, problem)
returnfloat(is_correct), confidence
Implement the two-player RL training loop:
classTwoPlayerRLTrainer:
"""
Train abstraction and solution generators together via RL.
"""def__init__(self, abstraction_model, solution_model, verifier):
self.abstraction_gen = AbstractionGenerator(abstraction_model)
self.solution_gen = SolutionGenerator(solution_model, verifier)
self.verifier = verifier
deftraining_step(self, problem, verifier, num_abstraction_samples=3):
"""
Single training step: generate abstractions and solutions, compute rewards.
Args:
problem: Problem to solve
verifier: Correctness verifier
num_abstraction_samples: Number of abstraction sets to try
Returns:
loss_abstractions: Loss for abstraction generator
loss_solution: Loss for solution generator
"""
all_losses_abs = []
all_losses_sol = []
best_solution_success = 0# Try multiple abstraction setsfor _ inrange(num_abstraction_samples):
# Generate abstractions
abstractions = self.abstraction_gen.generate_abstractions(problem)
# Generate solution using these abstractions
solution = self.solution_gen.solve_with_abstractions(problem, abstractions)
# Check correctness
is_correct, confidence = self.solution_gen.score_solution(solution, problem)
# Reward signal: solution success
solution_reward = is_correct
# Abstraction reward: same as solution (they share credit)# In practice, could use more sophisticated credit assignment
abstraction_reward = is_correct
# Loss for solution generator (maximize solution reward)
loss_sol = -torch.log(torch.tensor(solution_reward + 1e-6))
all_losses_sol.append(loss_sol)
# Loss for abstraction generator (maximize abstraction reward)# Make reward proportional to abstraction usefulness
abstraction_usefulness = 0.0for abs_item in abstractions:
abs_use = self.abstraction_gen.score_abstraction_usefulness(
abs_item,
problem,
is_correct
)
abstraction_usefulness += abs_use
abstraction_usefulness /= len(abstractions)
loss_abs = -torch.log(torch.tensor(abstraction_usefulness + 1e-6))
all_losses_abs.append(loss_abs)
best_solution_success = max(best_solution_success, is_correct)
# Average losses across samples
avg_loss_abs = torch.stack(all_losses_abs).mean()
avg_loss_sol = torch.stack(all_losses_sol).mean()
return avg_loss_abs, avg_loss_sol, best_solution_success
deftrain(self, problems, num_epochs=5):
"""
Train both generators on problem set.
Args:
problems: List of problems
num_epochs: Training epochs
Returns:
trained_models: (abstraction_model, solution_model)
"""
abs_optimizer = torch.optim.AdamW(
self.abstraction_gen.model.parameters(),
lr=1e-5
)
sol_optimizer = torch.optim.AdamW(
self.solution_gen.model.parameters(),
lr=1e-5
)
for epoch inrange(num_epochs):
total_loss_abs = 0
total_loss_sol = 0
total_success = 0for problem in problems:
# Forward pass
loss_abs, loss_sol, success = self.training_step(problem)
# Backward pass for abstractions
abs_optimizer.zero_grad()
loss_abs.backward()
abs_optimizer.step()
# Backward pass for solutions
sol_optimizer.zero_grad()
loss_sol.backward()
sol_optimizer.step()
total_loss_abs += loss_abs.item()
total_loss_sol += loss_sol.item()
total_success += success
print(f"Epoch {epoch+1}: "f"loss_abs={total_loss_abs/len(problems):.4f}, "f"loss_sol={total_loss_sol/len(problems):.4f}, "f"success_rate={total_success/len(problems):.2%}")
returnself.abstraction_gen.model, self.solution_gen.model
Implement allocation of computation at test time:
deftest_time_scaling_with_abstractions(
problem,
abstraction_gen,
solution_gen,
verifier,
compute_budget=100# Tokens or inference calls):
"""
At test time, decide whether to generate more abstractions or more solutions.
Args:
problem: Problem to solve
abstraction_gen: Trained abstraction generator
solution_gen: Trained solution generator
verifier: Correctness verifier
compute_budget: Total compute available
Returns:
best_solution: Best solution found within budget
"""# Split budget between abstraction generation and solution generation# Research finding: more abstractions often better than more solutions
num_abstractions = max(1, compute_budget // 10)
solutions_per_abstraction = max(1, (compute_budget - 10) // num_abstractions)
best_solution = None
best_score = 0# Generate multiple abstraction setsfor _ inrange(num_abstractions):
abstractions = abstraction_gen.generate_abstractions(problem)
# Generate multiple solutions for each abstraction setfor _ inrange(solutions_per_abstraction):
solution = solution_gen.solve_with_abstractions(problem, abstractions)
is_correct, _ = solution_gen.score_solution(solution, problem)
if is_correct > best_score:
best_solution = solution
best_score = is_correct
return best_solution
Practical Guidance
When to use RLAD:
Reasoning tasks where abstractions significantly reduce complexity (math, logic, coding)
Key allocation insight: At larger compute budgets, allocating more compute to abstraction generation yields better returns than generating more solutions with fixed abstractions. Trade-off curve:
Abstractions
Solutions Each
Accuracy
1
20
48%
2
10
52%
5
4
54%
10
2
55%
Common pitfalls:
Abstraction reward too weak: If abstractions don't receive clear reward signal, generator learns nothing. Verify that abstraction-less solutions have <50% success before training.
Two-player instability: Balancing two loss terms can oscillate. Use moving averages of losses; skip gradient steps if one player dominates.
Abstraction trivialism: If generator learns to emit generic statements ("be careful, think step by step"), it's not learning domain structure. Periodically inspect generated abstractions.
Verifier quality: Weak verifiers give conflicting reward signals. Validate verifier accuracy >90% before full training.
Integration checklist:
Validate solution generator baseline (without abstractions) on problem set
Generate 50 abstractions manually; verify they're concise and useful
Train abstraction generator on these examples
Run training step on 10 problems; monitor both loss curves
Periodically sample abstractions during training; verify quality
Evaluate solution generator with abstractions vs. without (measure accuracy gain)
Test compute allocation strategy (vary num_abstractions, measure accuracy)
