rlp-rl-pretraining-objective

name	rlp-rl-pretraining-objective
title	RLP: Reinforcement as a Pretraining Objective
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.01265
keywords	["pretraining","RL-training","reasoning","dense-reward","efficiency"]
description	Integrate reinforcement learning into the pretraining phase by measuring the utility of intermediate reasoning for predicting subsequent tokens. This approach generates dense reward signals during standard pretraining, enabling models to develop reasoning abilities earlier and with significant performance gains (19% improvement on 1.7B, 45% lift on 12B models).

RLP: RL-as-Pretraining for Early Reasoning Development

Current training pipelines treat reasoning and pretraining as separate phases: first train on raw text, then apply RL for reasoning. This separation is artificial. The insight behind RLP is that reasoning has immediate utility during pretraining itself—chain-of-thought reasoning helps predict the next token better than direct prediction. You can measure this utility (reward signal) and use it to guide pretraining, merging RL and next-token prediction into a unified objective.

Traditional next-token prediction treats all reasoning equally. RLP instead rewards reasoning tokens that increase prediction likelihood, creating a natural curriculum where the model learns to reason when it's beneficial and to skip reasoning when unnecessary.

Core Concept

RLP measures reward as the improvement in next-token likelihood when conditioning on sampled reasoning:

Reward = log P(next_token | context + reasoning) - log P(next_token | context)

This is a dense reward signal: every reasoning token gets immediate feedback on whether it actually helps prediction. This enables learning from raw pretraining data without external verifiers or gold reasoning traces.

Architecture Overview

• Base LLM: Standard transformer model
• Reasoning sampler: Generate intermediate reasoning (either sampled or guided)
• Reward computer: Measure log-likelihood improvement from reasoning
• Training objective: Weighted combination of reasoning utility and next-token prediction
• Scheduler: Control reasoning frequency over training (start high, adapt based on reward)

Implementation Steps

Start by implementing the reward computation. This is the core signal:

import torch
import torch.nn.functional as F

class ReasoningRewardComputer:
    """
    Compute reward for intermediate reasoning during pretraining.
    """
    def __init__(self, model, tokenizer, temperature=0.7):
        self.model = model
        self.tokenizer = tokenizer
        self.temperature = temperature

     ():
        
         torch.no_grad():
            
            outputs_no_reasoning = .model(context)
            logits_no_reasoning = outputs_no_reasoning.logits[:, -, :]  
            logprob_no_reasoning = F.log_softmax(logits_no_reasoning, dim=-)[
                (context.shape[]),
                next_token
            ]

            
            reasoning_input = torch.cat([context, reasoning], dim=)
            outputs_with_reasoning = .model(reasoning_input)
            logits_with_reasoning = outputs_with_reasoning.logits[:, -, :]
            logprob_with_reasoning = F.log_softmax(logits_with_reasoning, dim=-)[
                (reasoning_input.shape[]),
                next_token
            ]

        
        reward = logprob_with_reasoning - logprob_no_reasoning

         reward, logits_with_reasoning, logits_no_reasoning