pretraining-midtraining-rl-interplay

name	pretraining-midtraining-rl-interplay
title	On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.07783
keywords	["reinforcement learning","reasoning","pre-training","mid-training","capability development"]
description	Understand when RL genuinely expands reasoning beyond pre-training through controlled experiments on synthetic tasks. Discover that RL works best at the edge of competence and process rewards reduce hacking—critical for designing effective reasoning model training.

Overview

This work investigates whether RL in language models genuinely expands reasoning capabilities beyond what models acquire during initial training. Using synthetic reasoning tasks to isolate different training phases, the research reveals conditions under which RL generates real capability improvements versus when it merely optimizes existing knowledge.

When to Use

• Designing RL post-training curricula for reasoning models
• Understanding contribution of pre-training versus RL to capabilities
• Scenarios where process-level rewards matter for reasoning integrity
• Determining whether additional RL training is worthwhile
• Models where RL seems to plateau or hit diminishing returns

When NOT to Use

• Simple supervised fine-tuning scenarios
• Tasks where pre-training alone suffices
• Models with no room for RL improvements
• Situations where you need results immediately without careful analysis

Core Technique

Controlled experimental framework isolating training phase contributions:

# Analysis framework for training phase interactions
class TrainingPhaseAnalysis:
    def __init__(self):
        self.synthetic_tasks = SyntheticReasoningTasks()

    def study_rl_effectiveness(self, model, task_distribution):
        """
        Analyze when RL generates genuine capability improvements
        versus optimizing existing knowledge. Three key findings:
        1. RL needs adequate room for growth from pre-training
        2. RL targets "edge of competence" for effectiveness
        3. Generalization requires minimal but adequate pre-training exposure
        """
        results = {}

        # Dimension 1: Pre-training exposure level
        for exposure_level in [0.1, 0.3, 0.5, 0.7, 0.9]:
            model_pretrained = self.pretrain_with_exposure(
                task_distribution,
                exposure_level
            )

            
            model_rl = .apply_rl(model_pretrained)

            
            
            extrap_improvement = .measure_extrapolation_gain(
                model_pretrained,
                model_rl
            )

            results[] = extrap_improvement

         results

     ():
        
        performance_metrics = []

         task  tasks:
            
            base_perf = .evaluate_model(model, task)

            
            model_rl = .apply_focused_rl(model, task)
            rl_perf = .evaluate_model(model_rl, task)

            
            
            improvement = rl_perf - base_perf
            performance_metrics.append({
                : task,
                : base_perf,
                : rl_perf,
                : improvement
            })

         performance_metrics

     ():
        
        
        model_pretrain = .pretrain(dataset)
        perf_pretrain = .evaluate(model_pretrain)

        
        model_rl = .apply_rl(model_pretrain)
        perf_rl = .evaluate(model_rl)

        
        model_midtrain = .midtrain(model_pretrain, dataset)
        model_full = .apply_rl(model_midtrain)
        perf_full = .evaluate(model_full)

         {
            : perf_pretrain,
            : perf_rl,
            : perf_full
        }

     ():
        
        
        model_outcome = .train_with_outcome_rewards(model, tasks)
        perf_outcome = .evaluate(model_outcome)
        integrity_outcome = .measure_reasoning_integrity(model_outcome)

        
        model_process = .train_with_process_rewards(model, tasks)
        perf_process = .evaluate(model_process)
        integrity_process = .measure_reasoning_integrity(model_process)

         {
            : {
                : perf_outcome,
                : integrity_outcome
            },
            : {
                : perf_process,
                : integrity_process
            }
        }

     ():
        
        
        extrapolative_gen = .measure_extrapolative_generalization(
            model, contexts
        )

        
        contextual_gen = .measure_contextual_generalization(
            model, contexts
        )

         {
            : extrapolative_gen,
            : contextual_gen
        }