prl-process-reward-learning

name	prl-process-reward-learning
title	PRL: Process Reward Learning Improves LLMs' Reasoning Ability and Broadens the Reasoning Boundary
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2601.10201
keywords	["reasoning","process-rewards","RL","trajectory-supervision","entropy-regularization"]
description	Improves LLM reasoning by decomposing RL objectives into intermediate process rewards assigned to reasoning steps, improving both final accuracy and reasoning capacity without expensive Monte Carlo Tree Search.

Overview

Enhance LLM reasoning capabilities by providing reward signals at intermediate steps of the reasoning process rather than only at the end. This approach decomposes entropy-regularized RL objectives into granular process-level rewards that guide step-by-step improvement.

When to Use

• For complex reasoning tasks (math, logic, science, coding)
• When you want to improve pass@k performance
• For multi-step problem-solving where intermediate correctness matters
• When computational efficiency is important (vs. MCTS or expensive sampling)

When NOT to Use

• For single-step tasks without intermediate reasoning
• When intermediate ground truth is unavailable
• For real-time applications where reward computation adds latency
• For tasks where final-outcome-only feedback is sufficient

Key Technical Components

Process Reward Decomposition

Break down the global RL objective into step-level rewards.

# Process reward decomposition
class ProcessRewardLearner:
    def __init__(self, reference_model):
        self.reference_model = reference_model
        self.process_rewards = {}

    def decompose_objective(self, trajectory, final_reward):
        """Decompose end-reward into process rewards"""
        # Original objective: max R(trajectory) - KL(policy || reference)
        # Decomposed: sum of process rewards for each step

        trajectory_length = len(trajectory)
        base_reward = final_reward / trajectory_length

        process_rewards = []
        for step_idx, step in enumerate(trajectory):
            # Assign step-level reward
            step_reward = self.compute_step_reward(
                step,
                step_idx,
                trajectory,
                final_reward
            )
            process_rewards.append(step_reward)

        return process_rewards

     ():
        
        
        
        
        

        step_logprob = step[]
        reference_logprob = .reference_model.get_logprob(step[])

        kl_penalty = step_logprob - reference_logprob
        progress_bonus = .estimate_progress(idx, full_trajectory)

         progress_bonus -  * kl_penalty