| name | h2sd-hybrid-hindsight-self-distillation |
| version | 1.0.0 |
| description | Hybrid Hindsight Self-Distillation for Reinforcement Learning with Verifiable Rewards |
| trigger_words | ["h2sd","hybrid hindsight self-distillation","rlvr","on-policy self-distillation"] |
| arxiv_id | 2607.18955 |
H²SD: Hybrid Hindsight Self-Distillation
Overview
H²SD (Hybrid Hindsight Self-Distillation) is a framework that improves reinforcement learning with verifiable rewards (RLVR) by providing better token-level credit assignment through a hybrid approach to self-distillation.
Key Innovation
H²SD uses the teacher signal differently based on trajectory correctness:
- For successful trajectories: Uses teacher probabilities to modulate update magnitudes without changing direction
- For failed trajectories: Minimizes reverse KL divergence from student to teacher conditioned on reference hints
Implementation Steps
- Setup Base Model: Start with a language model suitable for your reasoning task
- Implement Trajectory Evaluation: Create a mechanism to verify if a reasoning trajectory is correct
- Create Teacher Conditioning:
- For successful trajectories: Condition teacher on student response + rephrasing instruction
- For failed trajectories: Condition teacher on reference hint + verified answer
- Apply Hybrid Distillation:
- Successful: Use teacher probabilities as magnitude weights for gradient updates
- Failed: Minimize reverse KL divergence to teacher distribution
- Train with RLVR: Combine with standard RLVR reward signals
Benefits
- Consistently outperforms RLVR, OPSD, and RLSD baselines
- Maintains stable optimization during training
- Provides explicit correction direction for failed reasoning
- Preserves generation efficiency
Use Cases
- Mathematical reasoning with LLMs
- Code generation with verification
- Complex multi-step reasoning tasks
- Any domain where verifiable rewards are available
References
