sage-self-hinting-rl

name	sage-self-hinting-rl
title	Self-Hinting Language Models Enhance Reinforcement Learning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.03143
keywords	["Reinforcement Learning","Training Instability","Reward Shaping","Curriculum Learning","GRPO"]
description	Inject privileged hints during GRPO training to reshape rollout distributions when advantage collapses occur, increasing outcome diversity without changing task rewards. Hints removed at deployment; policy automatically learns when to use hints via online refresh mechanism.

SAGE: Self-Hinting for Stable LLM Reinforcement Learning

Standard GRPO training struggles on difficult tasks with sparse rewards: rollout groups often receive identical rewards (all success or all failure), causing advantage calculations to collapse and learning to stall. SAGE addresses this by injecting hints during training—procedural guidance derived from reference solutions—that reshape the reward distribution to create outcome diversity within groups. Critically, hints only affect training; at deployment, they're removed entirely.

The key innovation is policy-dependent hint scheduling: hints activate automatically when advantage collapses, creating an effective curriculum that matches model capability. This contrasts with static curricula and avoids explicit difficulty estimation.

Core Concept

SAGE operates on two principles:

1. Privileged Supervision During Training: Hints (lightweight procedural guidance) reshape the rollout distribution to ensure mixed outcomes within each group, preventing advantage collapse.

2. Online Hint Refresh: The hint generator is periodically updated using the current policy, keeping hints calibrated to the learner's evolving capabilities.

This creates a dynamic curriculum without explicit difficulty labels or multi-stage training schedules.

Architecture Overview

• Hint Generator: Creates procedural guidance at variable strength levels using reference solutions
• Policy-Dependent Scheduler: Activates hints only when groups collapse (all same reward)
• GRPO Training Loop: Standard GRPO with conditional hint injection
• Online Refresh: Periodically regenerate hints using current policy predictions
• Evaluation & Removal: At inference, hints are set to empty strings (no privileged info)

Implementation

Step 1: Design Hint Generation

Create a system to generate hints of varying strength from reference solutions.

# Hint generation system
class HintGenerator:
    def __init__(self, model: str = "gpt-3.5-turbo"):
        """Generate procedural hints from solutions."""
        self.model = model

     () -> [, ]:
        
        hints = {}

         strength  strength_levels:
             strength == :
                hints[strength] =   
             strength < :
                
                hint_prompt = 
                hint = .model.generate(hint_prompt, max_tokens=)
                hints[strength] = hint

             strength < :
                
                hint_prompt = 
                hint = .model.generate(hint_prompt, max_tokens=)
                hints[strength] = hint

            :
                
                hint = reference_solution[:(reference_solution) * ]
                hints[strength] = hint

         hints

     () -> [, [, ]]:
        
        all_hints = {}

         prompt, solution  (prompts, reference_solutions):
            sample_id = (prompt) % 
            all_hints[sample_id] = .generate_hints_from_solution(
                prompt,
                solution,
                strength_levels
            )

         all_hints