| name | let-it-calm-annealed-decoding |
| title | Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learning |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2510.05251 |
| keywords | ["Decoding Strategy","Temperature Scheduling","Reinforcement Learning","Exploration","Training Stability"] |
| description | Use dynamic temperature scheduling that explores early (high temperature) and exploits late (low temperature) during generation, improving sample efficiency in RL with verifiable rewards. |
Technique: Exploratory-then-Exploitative Temperature Scheduling
Exploration isn't equally valuable at all points during sequence generation. Early tokens define semantic direction—high uncertainty here drives diverse meaningful outputs. Later tokens fill details where exploration adds noise. Let it Calm (EAD) implements this insight through temperature scheduling that starts warm (explore early) and cools down (exploit late).
This simple but effective approach improves sample efficiency in RLVR (RL with verifiable rewards) by encouraging semantic diversity when it matters while maintaining training stability through low-entropy later generation.
Core Concept
Exploratory Annealed Decoding operates through three mechanisms:
-
Dynamic Temperature Schedule: Start high, decay over sequence length τₜ = max{τ_init - e^(t/d), τ_min}
-
Global-Step Awareness: Adjust decay rate based on training progress (longer responses later need slower decay)
-
Truncated Importance Sampling: Correct off-policy issues from aggressive annealing
Architecture Overview
- Initial Temperature: Start with τ > 1.0 for exploration
- Time-Dependent Decay: Linear-exponential schedule over token positions
- Training-Aware Adaptation: Slower decay as model improves
- IS Correction: Prevent training instability from distribution mismatch
- RL Integration: Plug into GRPO, DAPO, or other RLVR algorithms
Implementation Steps
Implement the dynamic temperature schedule.
def get_temperature_schedule(sequence_length, global_step,
tau_init=1.5, tau_min=0.5, decay_rate=0.1):
"""
Compute temperature for each timestep in generation.
Args:
sequence_length: Length of sequence being generated
global_step: Current training step
tau_init: Initial temperature (usually > 1.0)
tau_min: Minimum temperature
decay_rate: Decay rate parameter
Returns:
temperatures: Array of temperatures for each token position
"""
import numpy as np
temperatures = []
for t (sequence_length):
decay = np.exp(-t / (decay_rate * sequence_length))
tau_t = tau_min + (tau_init - tau_min) * decay
global_step > :
progress_factor = (, (global_step / ) ** )
tau_t = tau_min + (tau_t - tau_min) * progress_factor
temperatures.append(tau_t)
np.array(temperatures)
