dsdr-dual-scale-diversity-regularization

name	dsdr-dual-scale-diversity-regularization
title	DSDR: Dual-Scale Diversity Regularization for LLM Reasoning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.19895
keywords	["reinforcement learning","reasoning diversity","exploration","LLM training","regularization"]
description	Improve LLM reasoning by promoting diversity at both trajectory and token levels simultaneously. Global (trajectory) scale rewards distinct correct solutions; local (token) scale applies entropy regularization per decision point. Dual-scale approach couples these via diversity-weighted allocation: solutions that are globally more distinctive receive stronger local regularization, focusing exploration where it matters most among underexplored correct modes.

DSDR: Multi-Scale Diversity Promotion for Stable Reasoning Exploration

Language models trained with reinforcement learning on reasoning tasks often collapse onto narrow solution modes, missing diverse valid reasoning paths. Standard approaches either reward all correct solutions equally (missing that some are more valuable) or apply uniform entropy regularization (missing that not all decision points need exploration). The challenge is discovering which correct modes are underexplored and focusing exploration effort there.

Two key insights: (1) not all correct modes are equally valuable for downstream learning, (2) exploration effort should be concentrated at decision points that distinguish underexplored modes. Existing methods treat diversity at single scales, missing this multi-scale structure.

Core Concept

DSDR operates at two complementary scales:

Global (Trajectory) Scale: Promote diversity among correct reasoning paths by assigning higher rewards to distinct solutions. Use semantic diversity (embedding differences) and formula diversity (unique mathematical expressions) to measure distinctness.

Local (Token) Scale: Apply length-invariant entropy regularization within correct trajectories to prevent overconfidence at individual decision points.

Coupling Mechanism: Link scales by letting global distinctiveness guide local regularization strength. Solutions that are globally more distinctive receive stronger local entropy penalties, directing exploration toward decision points that matter for underexplored modes.

Architecture Overview

• Semantic Diversity Scorer: Embed solutions and compute pairwise cosine distances in semantic space
• Formula Diversity Tracker: Count unique mathematical expressions across correct solutions
• Global Reward Augmenter: Increase rewards for semantically/formulaically distinct solutions
• Local Entropy Computer: Compute per-token entropy within each trajectory
• Coupling Weight: Compute per-solution diversity weight (based on global distinctiveness) and scale local regularization accordingly
• Normalized Loss: Avoid length bias by averaging per-token, then scaling by diversity weight

Implementation

Compute semantic and formula diversity among correct solutions:

def compute_solution_diversity(correct_solutions):
    """
    Measure diversity of correct solutions using semantic and formula metrics.
    correct_solutions: list of (response_text, correctness_flag)
    Returns: diversity_scores dict
    """
     sklearn.metrics.pairwise  cosine_distances
     re

    
    texts = [sol[]  sol  correct_solutions  sol[]]

    
    embeddings = embed_texts(texts)  
    semantic_distances = cosine_distances(embeddings)
    avg_semantic_diversity = semantic_distances.mean()

    
    formulas = []
     text  texts:
        
        found_formulas = re.findall(, text)
        formulas.extend(found_formulas)

    unique_formulas = ((formulas))
    total_formulas = (formulas)
    formula_diversity = unique_formulas / (total_formulas, )

     {
        : avg_semantic_diversity,
        : formula_diversity,
        : (avg_semantic_diversity + formula_diversity) / 
    }