repro-process-reward

name	repro-process-reward
title	Rectifying LLM Thought from Lens of Optimization: Process-Level Rewards
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.01925
keywords	["reinforcement-learning","reasoning-optimization","process-rewards","llm-alignment","chain-of-thought"]
description	Dual-scoring mechanism (Magnitude and Stability scores) enabling process-level rewards that penalize inefficient reasoning patterns like overthinking and backtracking without trained reward models. Improve reasoning efficiency in RL without additional supervision.

Summary

RePro (Rectifying Process-level Reward) frames chain-of-thought reasoning as an optimization process and develops a dual-scoring mechanism to improve LLM reasoning during RL training. The method computes Magnitude Score (optimization intensity) and Stability Score (smoothness), combined into process-level rewards that penalize inefficient patterns while incentivizing efficient reasoning steps.

Core Technique

Surrogate Objective Function: Rather than evaluating reasoning directly, compute a surrogate measuring the model's confidence in ground truth via perplexity:

confidence_t = -log p(ground_truth_token | context_t)

This provides a continuous signal throughout reasoning about whether the model is on the right track.

Magnitude Score: Quantifies optimization intensity:

magnitude = |confidence_0 - confidence_T| / T

High magnitude indicates strong improvement or degradation; moderate magnitude indicates steady progress.

Stability Score: Assesses smoothness of reasoning trajectory:

stability = 1 - (variance of confidence across steps) / mean(confidence)

High stability indicates consistent, directed reasoning; low stability indicates zigzagging.

Process-Level Reward: Combine both scores:

reward_t = magnitude * stability - λ_overthink * overthinking_penalty

Implementation

Perplexity tracking: At each reasoning step, compute cross-entropy loss with ground truth:

step_loss = cross_entropy(logits_t, target)
confidence_t = -log(softmax(logits_t)[target])

Score computation:

magnitudes = [abs(confidence[i+1] - confidence[i]) for i in range(len(confidence)-1)]
magnitude_score = mean(magnitudes) / max(magnitudes)

variance = var(confidence)
stability_score = 1 - (variance / (mean(confidence) + ε))

Reward integration: In your RL pipeline:

process_reward = magnitude_score * stability_score
# Add to existing RL loss without extra models
total_reward = sequence_reward +  * process_reward