lspo-length-aware-dynamic-sampling

name	lspo-length-aware-dynamic-sampling
title	LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.01459
keywords	["Reinforcement Learning","LLM Training","Policy Optimization","Response Length Filtering","Reasoning Models"]
description	Filter training samples by response length to identify high-confidence correct solutions and complex problems, improving sample efficiency in LLM reasoning RL without additional metrics.

Technique: Length-Aware Dynamic Sampling for Efficient RL Training

Large language models trained on reasoning tasks generate responses of varying lengths. Existing approaches treat all responses equally, but response length contains useful training signal: short responses indicate high-confidence solutions, while long responses reveal problems that required substantial effort. LSPO exploits this signal to filter training data more efficiently.

The core insight is that intermediate-length responses lack the signal value of either extreme. Short responses demonstrate correct reasoning with minimal deliberation (the ideal case). Long responses show where models struggle and need reinforcement. Responses of moderate length neither demonstrate confidence nor provide clear learning opportunities.

Core Concept

LSPO implements a filtering strategy based on response length percentiles during each RL training iteration. The algorithm retains the shortest 30% of responses (high-confidence cases) and the longest 65-95% of responses (difficult cases requiring effort), filtering out the middle 35-65% as uninformative.

The percentile thresholds are computed dynamically from the current batch rather than fixed globally. This ensures the filtering adapts as the model's response distribution evolves during training—early in training when responses cluster in a narrow range, the percentiles shift automatically to maintain the retention ratio.

Architecture Overview

• Input: A batch of (prompt, response) pairs from an LLM rollout
• Filtering Stage: Compute response length for each example, determine 30th and 65th-95th percentile boundaries
• Selection: Retain responses shorter than 30th percentile and longer than dynamic threshold
• Output: Filtered samples fed into standard RL algorithms (GRPO, DAPO, PPO)
• Integration: Works as a preprocessing layer before policy gradient computation

Implementation Steps

Initialize the filtering parameters and generate a batch of responses. The filtering operates on response lengths computed as token count (typically using the model's tokenizer).

def lspo_filter(responses, prompts, retention_short=0.30, retention_long=0.30):
    """
    Filter responses by length using LSPO strategy.

    Args:
        responses: List of generated response strings
        prompts: List of corresponding prompts
        retention_short: Fraction of shortest responses to keep (default 0.30)
        retention_long: Fraction of longest responses to keep (default 0.30)

    Returns:
        filtered_indices: Boolean array of samples to retain
    """
    lengths = np.array([(r.split())  r  responses])

    
    short_threshold = np.percentile(lengths, retention_short * )
    long_threshold = np.percentile(lengths, ( - retention_long) * )

    
    keep_mask = (lengths <= short_threshold) | (lengths >= long_threshold)

     keep_mask