low-prob-exploration

name	low-prob-exploration
title	Low-Probability Tokens Sustain Exploration in Reasoning RL
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.03222
keywords	["reinforcement-learning","exploration","entropy-regularization","reasoning","token-filtering"]
description	Preserve reasoning diversity in RL by protecting low-probability tokens that represent novel thinking paths. Trigger: maintain exploration capability during RL training while avoiding entropy collapse on reasoning tasks.

Low-Probability Tokens Sustain Exploration

Core Concept

During RLVR (RL with Verifiable Rewards) training on reasoning tasks, models suffer from entropy collapse—the policy converges too tightly to common answer patterns, eliminating valuable exploratory reasoning. Low-probability tokens (rare in pretraining) represent genuine thinking diversity but are systematically eliminated during RL. This skill introduces Low-probability Regularization (Lp-Reg): create a proxy distribution filtering noise while protecting "reasoning sparks"—low-probability tokens that enable novel solution paths.

The key insight: Not all entropy is useful exploration; distinguish signal-carrying low-probability tokens from noise, and protect only the signal.

Architecture Overview

• Reasoning Sparks Identification: Detect low-probability tokens that carry reasoning value
• Proxy Distribution Creation: Filter out noise while preserving signal
• Soft Regularization: KL divergence-based entropy maintenance targeted to valuable tokens
• Stable On-Policy RL: Enable continuous scaling without entropy collapse
• Token-Level Analysis: Fine-grained understanding of what enables diverse reasoning

Implementation Steps

1. Identify Reasoning Sparks

Analyze which low-probability tokens contribute to diverse reasoning paths.

class ReasoningSparkDetector:
    """
    Identify low-probability tokens that enable novel reasoning.
    """
    def __init__(self, model, reference_model):
        self.model = model
        self.ref_model = reference_model  # Pretrained baseline

    def identify_reasoning_sparks(self, reasoning_traces, solutions, correctness):
        """
        Determine which low-probability tokens lead to correct solutions.

        Args:
            reasoning_traces: List of reasoning token sequences
            solutions: Corresponding final answers
            correctness: Boolean list of correctness

        Returns:
            Set of token IDs that represent reasoning sparks
        """
        spark_candidates = {}

        for trace, solution, is_correct  (reasoning_traces, solutions, correctness):
            tokens = tokenize(trace)

             token_id  tokens:
                token_prob_pretrain = .ref_model.get_token_prob(token_id)

                
                 token_prob_pretrain < :
                     token_id   spark_candidates:
                        spark_candidates[token_id] = {
                            : ,
                            : ,
                            : []
                        }

                    spark_candidates[token_id][] += 
                    spark_candidates[token_id][].append(solution)

                     is_correct:
                        spark_candidates[token_id][] += 

        
        reasoning_sparks = ()
         token_id, stats  spark_candidates.items():
            correctness_rate = stats[] / (stats[] + )

            
             correctness_rate >   stats[] > :
                reasoning_sparks.add(token_id)

         reasoning_sparks