Overcome the shallow exploration trap by explicitly rewarding longer reasoning sequences when models fail to solve problems. Use length-incentivized exploration to enable deeper chain-of-thought reasoning, achieving better test-time scaling and improved generalization across in-domain and out-of-domain tasks.
Instrucciones de origen · Vista previa de solo lectura
name
length-incentivized-exploration
title
Think Longer to Explore Deeper: Length-Incentivized RL
version
0.0.2
engine
skillxiv-v0.0.2-claude-opus-4.6
license
MIT
url
https://arxiv.org/abs/2602.11748
keywords
["Reinforcement Learning","Chain of Thought","Exploration","Length Reward","Test-Time Scaling"]
description
Overcome the shallow exploration trap by explicitly rewarding longer reasoning sequences when models fail to solve problems. Use length-incentivized exploration to enable deeper chain-of-thought reasoning, achieving better test-time scaling and improved generalization across in-domain and out-of-domain tasks.
Think Longer to Explore Deeper: Length-Incentivized RL
Problem Context
Language models naturally prefer generating shorter outputs due to exponential decay in sampling probabilities—tokens at the beginning have much higher probability of being selected than later tokens. However, solving difficult reasoning tasks requires extended reasoning chains. This creates a fundamental tension: models that naturally stop early miss opportunities for deeper exploration, even when they fail on initial attempts. Standard RL training does not incentivize this behavior change.
Core Concept
Length-Incentivized Exploration (LIE) addresses this by introducing two reward signals:
Length Reward (R_len): Explicitly reward extending reasoning length when the model fails to solve a problem immediately
Redundancy Penalty (R_red): Penalize repetitive content to ensure extensions explore new reasoning paths rather than filling space
Together, these signals create a curriculum where models learn to extend thinking when uncertain, while still avoiding meaningless output expansion.
Architecture Overview
Length detection: Identify when model's initial response fails to solve the problem
Adaptive length reward: Scale reward based on how much model extended reasoning beyond baseline
Redundancy detection: Measure token repetition and self-similarity in extended sequences
Combined objective: Length reward - redundancy penalty in GRPO framework
"""
Args:
failure_detector: Function that returns True if response fails the task
baseline_length_percentile: Percentile of response lengths to use as baseline
"""
self
self
self
def
detect_failure
self, response: str, task: str
bool
"""Check if response fails the task."""
return
self
def
measure_length
self, response: str
int
"""Measure reasoning length in tokens (approximate: words)."""
return
len
def
analyze_response
self,
response: str,
task: str
Dict
str
float
"""
Analyze a single response.
Returns:
failed: True if task failed
length: Number of tokens/words
extended: Whether response extended beyond typical length
"""
self
self
self
# Compute baseline length from observed distribution
if
len
self
10
self
self
else
# Initialize with first response
return
'failed'
'length'
'baseline_length'
'extended'
def
get_baseline_length
self
float
"""Get current baseline length estimate."""
if
len
self
0
return
self
self
return
100.0
# Default baseline
Step 2: Compute length reward
Reward extending reasoning when models fail initially.
classLengthRewardComputer:
"""Compute length-based rewards for exploration."""def__init__(
self,
length_reward_scale: float = 0.1,
min_extension_tokens: int = 10):
"""
Args:
length_reward_scale: Scale factor for length reward
min_extension_tokens: Minimum extension to count as exploration
"""self.length_reward_scale = length_reward_scale
self.min_extension_tokens = min_extension_tokens
defcompute_length_reward(
self,
response: str,
failed: bool,
baseline_length: float,
max_length: float = 2000.0) -> float:
"""
Compute length reward based on failure and extension.
Reward logic:
- If task succeeded: no length bonus (task reward is enough)
- If task failed AND response is short: reward extension
- If task failed AND response already long: slight reward for trying longer
"""
response_length = len(response.split())
ifnot failed:
# Task succeeded; length reward not neededreturn0.0# Task failed; reward if model extended reasoning
extension_length = max(0, response_length - baseline_length)
if extension_length < self.min_extension_tokens:
# Minimal extension; no rewardreturn0.0# Reward proportional to extension, capped at max_length
normalized_extension = min(extension_length, max_length - baseline_length)
length_reward = self.length_reward_scale * (
normalized_extension / max(1.0, max_length - baseline_length)
)
return length_reward
Step 3: Detect and penalize redundancy
Penalize repetitive output to ensure true exploration.
classRedundancyPenaltyComputer:
"""Compute penalties for redundant/repetitive content."""def__init__(
self,
redundancy_penalty_scale: float = 0.05,
ngram_sizes: list = [1, 2, 3]
):
"""
Args:
redundancy_penalty_scale: Scale for redundancy penalty
ngram_sizes: N-gram sizes to check for repetition
"""self.redundancy_penalty_scale = redundancy_penalty_scale
self.ngram_sizes = ngram_sizes
defextract_ngrams(self, tokens: list, n: int) -> Dict[tuple, int]:
"""Extract n-gram frequencies."""
ngrams = {}
for i inrange(len(tokens) - n + 1):
gram = tuple(tokens[i:i + n])
ngrams[gram] = ngrams.get(gram, 0) + 1return ngrams
defcompute_redundancy_score(
self,
response: str,
baseline_length: float) -> float:
"""
Measure redundancy in response extension.
Focus on extended portion to detect padding vs. exploration.
"""
tokens = response.split()
response_length = len(tokens)
# Only analyze extended portion
baseline_tokens = int(baseline_length)
if response_length <= baseline_tokens:
return0.0# No extension; no redundancy penalty
extended_portion = tokens[baseline_tokens:]
iflen(extended_portion) < 5:
return0.0# Too short to meaningfully analyze# Compute repetition scores for different n-grams
total_redundancy = 0.0for n inself.ngram_sizes:
if n > len(extended_portion):
continue
ngrams = self.extract_ngrams(extended_portion, n)
# Count repeated n-grams
repeated_count = sum(count - 1for count in ngrams.values() if count > 1)
total_ngrams = len(extended_portion) - n + 1if total_ngrams > 0:
redundancy_ratio = repeated_count / total_ngrams
total_redundancy += redundancy_ratio
# Average over n-gram sizes
avg_redundancy = total_redundancy / len(self.ngram_sizes)
returnself.redundancy_penalty_scale * avg_redundancy
defcompute_redundancy_penalty(
self,
response: str,
baseline_length: float,
failed: bool) -> float:
"""
Compute final redundancy penalty.
Only apply to extended, failed responses.
"""
extension_length = len(response.split()) - baseline_length
if extension_length < 10ornot failed:
return0.0# Don't penalize if minimal extension or already succeededreturnself.compute_redundancy_score(response, baseline_length)
Step 4: Combine length and redundancy into composite reward
