uniqueness-aware-rl

name	uniqueness-aware-rl
title	Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2601.08763
keywords	["exploration","RL","creative-problem-solving","diversity-reward","reasoning"]
description	Improves LLM reasoning by rewarding correct solutions that exhibit rare high-level strategies, preventing exploration collapse and discovering more diverse solution approaches across mathematics, physics, and medical reasoning.

Overview

Enhance exploration in LLM reasoning by rewarding correct solutions based on their rarity of approach. Rather than treating all correct solutions equally, provide higher rewards for solutions using novel high-level strategies, preventing models from converging to single solution patterns.

When to Use

• For reasoning tasks where multiple solution approaches exist
• When you want to maximize the diversity of reasoning strategies learned
• For improving pass@k metrics by exploring different solution paths
• For creative problem-solving where approach diversity is valuable

When NOT to Use

• For tasks with unique optimal solutions
• When solution diversity doesn't matter for the application
• For single-solution problems
• When computational overhead of strategy analysis is unacceptable

Key Technical Components

Solution Strategy Clustering

Group solutions by high-level reasoning approach, ignoring surface-level variation.

# Strategy clustering for exploration diversity
class SolutionStrategyClusterer:
    def __init__(self, strategy_classifier_model):
        self.classifier = strategy_classifier_model

    def extract_strategy(self, solution_text, problem):
        """Identify high-level strategy from solution"""
        prompt = f"""
        Analyze this {problem['domain']} solution:
        {solution_text}

        What high-level strategy does it use? (e.g., algebraic, graphical, numerical, recursive)
        Ignore surface-level variations in expression.
        """
        strategy = self.classifier.classify(prompt)
        return strategy

    def cluster_solutions(self, solutions, problem):
        """Group solutions by strategy"""
        strategy_groups = {}

        for solution  solutions:
            strategy = .extract_strategy(solution[], problem)

             strategy   strategy_groups:
                strategy_groups[strategy] = []

            strategy_groups[strategy].append(solution)

         strategy_groups

     ():
        
        strategy_counts = {}
         s  all_strategies:
            strategy_counts[s] = strategy_counts.get(s, ) + 

        
        frequency = strategy_counts.get(strategy, ) / (all_strategies)
        rarity =  / (frequency + )

        
        normalized_rarity =  / (rarity + )  

         normalized_rarity