| name | lspo-length-aware-dynamic-sampling |
| title | LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoning |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2510.01459 |
| keywords | ["Reinforcement Learning","LLM Training","Policy Optimization","Response Length Filtering","Reasoning Models"] |
| description | Filter training samples by response length to identify high-confidence correct solutions and complex problems, improving sample efficiency in LLM reasoning RL without additional metrics. |
Technique: Length-Aware Dynamic Sampling for Efficient RL Training
Large language models trained on reasoning tasks generate responses of varying lengths. Existing approaches treat all responses equally, but response length contains useful training signal: short responses indicate high-confidence solutions, while long responses reveal problems that required substantial effort. LSPO exploits this signal to filter training data more efficiently.
The core insight is that intermediate-length responses lack the signal value of either extreme. Short responses demonstrate correct reasoning with minimal deliberation (the ideal case). Long responses show where models struggle and need reinforcement. Responses of moderate length neither demonstrate confidence nor provide clear learning opportunities.
Core Concept
LSPO implements a filtering strategy based on response length percentiles during each RL training iteration. The algorithm retains the shortest 30% of responses (high-confidence cases) and the longest 65-95% of responses (difficult cases requiring effort), filtering out the middle 35-65% as uninformative.
The percentile thresholds are computed dynamically from the current batch rather than fixed globally. This ensures the filtering adapts as the model's response distribution evolves during training—early in training when responses cluster in a narrow range, the percentiles shift automatically to maintain the retention ratio.
Architecture Overview
- Input: A batch of (prompt, response) pairs from an LLM rollout
- Filtering Stage: Compute response length for each example, determine 30th and 65th-95th percentile boundaries
- Selection: Retain responses shorter than 30th percentile and longer than dynamic threshold
- Output: Filtered samples fed into standard RL algorithms (GRPO, DAPO, PPO)
- Integration: Works as a preprocessing layer before policy gradient computation
Implementation Steps
Initialize the filtering parameters and generate a batch of responses. The filtering operates on response lengths computed as token count (typically using the model's tokenizer).
def lspo_filter(responses, prompts, retention_short=0.30, retention_long=0.30):
"""
Filter responses by length using LSPO strategy.
Args:
responses: List of generated response strings
prompts: List of corresponding prompts
retention_short: Fraction of shortest responses to keep (default 0.30)
retention_long: Fraction of longest responses to keep (default 0.30)
Returns:
filtered_indices: Boolean array of samples to retain
"""
lengths = np.array([(r.split()) r responses])
short_threshold = np.percentile(lengths, retention_short * )
long_threshold = np.percentile(lengths, ( - retention_long) * )
keep_mask = (lengths <= short_threshold) | (lengths >= long_threshold)
keep_mask
