| name | pretraining-midtraining-rl-interplay |
| title | On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2512.07783 |
| keywords | ["reinforcement learning","reasoning","pre-training","mid-training","capability development"] |
| description | Understand when RL genuinely expands reasoning beyond pre-training through controlled experiments on synthetic tasks. Discover that RL works best at the edge of competence and process rewards reduce hacking—critical for designing effective reasoning model training. |
Overview
This work investigates whether RL in language models genuinely expands reasoning capabilities beyond what models acquire during initial training. Using synthetic reasoning tasks to isolate different training phases, the research reveals conditions under which RL generates real capability improvements versus when it merely optimizes existing knowledge.
When to Use
- Designing RL post-training curricula for reasoning models
- Understanding contribution of pre-training versus RL to capabilities
- Scenarios where process-level rewards matter for reasoning integrity
- Determining whether additional RL training is worthwhile
- Models where RL seems to plateau or hit diminishing returns
When NOT to Use
- Simple supervised fine-tuning scenarios
- Tasks where pre-training alone suffices
- Models with no room for RL improvements
- Situations where you need results immediately without careful analysis
Core Technique
Controlled experimental framework isolating training phase contributions:
class TrainingPhaseAnalysis:
def __init__(self):
self.synthetic_tasks = SyntheticReasoningTasks()
def study_rl_effectiveness(self, model, task_distribution):
"""
Analyze when RL generates genuine capability improvements
versus optimizing existing knowledge. Three key findings:
1. RL needs adequate room for growth from pre-training
2. RL targets "edge of competence" for effectiveness
3. Generalization requires minimal but adequate pre-training exposure
"""
results = {}
for exposure_level in [0.1, 0.3, 0.5, 0.7, 0.9]:
model_pretrained = self.pretrain_with_exposure(
task_distribution,
exposure_level
)
model_rl = .apply_rl(model_pretrained)
extrap_improvement = .measure_extrapolation_gain(
model_pretrained,
model_rl
)
results[] = extrap_improvement
results
():
performance_metrics = []
task tasks:
base_perf = .evaluate_model(model, task)
model_rl = .apply_focused_rl(model, task)
rl_perf = .evaluate_model(model_rl, task)
improvement = rl_perf - base_perf
performance_metrics.append({
: task,
: base_perf,
: rl_perf,
: improvement
})
performance_metrics
():
model_pretrain = .pretrain(dataset)
perf_pretrain = .evaluate(model_pretrain)
model_rl = .apply_rl(model_pretrain)
perf_rl = .evaluate(model_rl)
model_midtrain = .midtrain(model_pretrain, dataset)
model_full = .apply_rl(model_midtrain)
perf_full = .evaluate(model_full)
{
: perf_pretrain,
: perf_rl,
: perf_full
}
():
model_outcome = .train_with_outcome_rewards(model, tasks)
perf_outcome = .evaluate(model_outcome)
integrity_outcome = .measure_reasoning_integrity(model_outcome)
model_process = .train_with_process_rewards(model, tasks)
perf_process = .evaluate(model_process)
integrity_process = .measure_reasoning_integrity(model_process)
{
: {
: perf_outcome,
: integrity_outcome
},
: {
: perf_process,
: integrity_process
}
}
():
extrapolative_gen = .measure_extrapolative_generalization(
model, contexts
)
contextual_gen = .measure_contextual_generalization(
model, contexts
)
{
: extrapolative_gen,
: contextual_gen
}
