Enable models to autonomously improve on target tasks during inference via test-time curricula (TTC-RL). Automatically select task-relevant training examples and apply RL to continue learning. Achieve 1.8x improvement on AIME25 math benchmarks and 2.1x on CodeElo competitive coding by learning task-specific skills at test time without human curation.
Enable models to autonomously improve on target tasks during inference via test-time curricula (TTC-RL). Automatically select task-relevant training examples and apply RL to continue learning. Achieve 1.8x improvement on AIME25 math benchmarks and 2.1x on CodeElo competitive coding by learning task-specific skills at test time without human curation.
Learning on the Job: Test-Time Curricula for Targeted RL
Core Concept
Traditional training assumes the task distribution is fixed. Learning on the Job (LotJ) flips this assumption: during inference on a target task, the model autonomously assembles a task-specific curriculum from available training data and continues learning via RL. This enables dramatic improvements (1.8-2.1x) on challenging benchmarks by specializing to each problem's characteristics.
Architecture Overview
Test-Time Curriculum Assembly: Automatically select relevant training examples from large pools without human curation
Task-Specific Skill Development: Apply RL to continue training on examples most relevant to target task
Continual Inference-Time Learning: Extend test-time scaling paradigm beyond planning to actual policy updates
Multi-Domain Generalization: Works across mathematical reasoning (AIME25), competitive coding (CodeElo), diverse task types
Stateful Learning: Maintain learned skills across examples within same inference session
Implementation Steps
1. Test-Time Curriculum Assembly
Automatically select which training examples to study for each target task.
classTestTimeCurriculumAssembler:
def__init__(self, training_pool, embedding_model='gpt-4.1'):
self.training_pool = training_pool # All available training examplesself.embedder = embedding_model
self.selected_curriculum = []
defassemble_curriculum(self, target_task, curriculum_size=100, budget=50):
"""
Assemble task-specific curriculum from training pool.
Args:
target_task: Target problem to solve
curriculum_size: Max examples in curriculum
budget: RL training steps available
"""# Step 1: Embed target task
target_embedding = self.embedder.embed(target_task)
candidate_examples = []
example .training_pool:
example_embedding = .embedder.embed(example[])
similarity = cosine_similarity(target_embedding, example_embedding)
candidate_examples.append((similarity, example))
candidate_examples.sort(reverse=)
selected = []
selected_solutions = ()
similarity, example candidate_examples[:curriculum_size * ]:
solution_pattern = ._extract_pattern(example[])
solution_pattern selected_solutions:
selected.append(example)
selected_solutions.add(solution_pattern)
(selected) >= curriculum_size:
.selected_curriculum = ._order_by_difficulty(selected, target_task)
.selected_curriculum
():
keywords = [, , , , ]
patterns = [kw kw keywords kw solution.lower()]
(patterns) patterns (,)
():
difficulty_scores = []
example examples:
difficulty = (example[].split()) /
similarity_to_target = cosine_similarity(
.embedder.embed(example[]),
.embedder.embed(target_task)
)
score = difficulty * ( - similarity_to_target)
difficulty_scores.append((score, example))
difficulty_scores.sort()
[ex _, ex difficulty_scores]
# Step 2: Retrieve relevant training examples by similarity
for
in
self
self
'problem'
# Sort by similarity (relevance)
True
# Step 3: Select diverse subset
# Avoid redundancy: pick examples covering different solution patterns
set
for
in
2
self
'solution'
if
not
in
if
len
break
# Step 4: Order curriculum by difficulty (easy → hard)
Curriculum Assembly: Similarity + diversity is key. High-similarity examples teach task-specific techniques; diverse examples prevent overfitting to specific solution patterns.
RL Budget: 50 steps (~5-10 minutes on modest hardware) yields significant improvements. More steps show diminishing returns (logarithmic scaling).
Training Pool: Larger pools (10K+ examples) enable better curriculum selection. Pool should cover diverse solution approaches for target domain.
Statefulness: Model changes persist across problems within a session. Consider resetting between independent tasks to avoid negative transfer.
When to Use / When NOT to Use
Use When:
Challenging benchmark problems (math, coding) where specialization helps
Test-time compute budget available (50-500 RL steps)
Training pool of relevant examples exists
Each problem benefits from learning task-specific skills
Domains where test-time training causes negative transfer
Scenarios lacking relevant training examples
Reference
This skill synthesizes findings from "Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learning" (arXiv:2510.04786). Test-time learning extends scaling paradigms beyond planning to active policy improvement during inference.
