Compress verbose reasoning by conditioning the same model on a conciseness instruction to create a teacher. Minimize KL divergence between student outputs and concise teacher without ground truth, enabling implicit reward learning that improves accuracy while reducing tokens.
Compress verbose reasoning by conditioning the same model on a conciseness instruction to create a teacher. Minimize KL divergence between student outputs and concise teacher without ground truth, enabling implicit reward learning that improves accuracy while reducing tokens.
On-Policy Self-Distillation for Reasoning Compression
Long reasoning chains improve model accuracy but consume significant compute and tokens. Traditional compression requires ground-truth short reasoning or external reward models. OPSDC takes a simpler approach: instruct the same model to be concise, then teach it to match that concise version without explicit supervision. The resulting compression naturally adapts to problem difficulty—easy problems receive strong pressure to be brief; hard problems preserve reasoning length.
The core innovation elegantly combines on-policy learning, reverse KL divergence for mode-seeking behavior, and implicit difficulty adaptation. The method simultaneously improves accuracy (by eliminating verbose error-prone reasoning) and reduces tokens.
Core Concept
OPSDC operates through four coordinated principles:
Same-Model Teacher: Condition target model on "be concise" instruction to generate a teacher; no external oracle needed
On-Policy Distillation: Train on student-generated rollouts (on-policy) to avoid distribution shift common in offline distillation
Mode-Seeking Behavior: Use reverse KL divergence to concentrate probability on teacher's preferred reasoning paths
Implicit Difficulty Scaling: Hard problems naturally receive weak compression pressure because even the concise teacher needs extended reasoning
Architecture Overview
Student Model: Base reasoning model generating trajectories
Teacher Model: Same model prompted with "be concise" instruction
On-Policy Sampling: Collect rollouts from student under current policy
KL Divergence: Compute reverse KL from teacher to student for mode-seeking
Loss Scaling: Implicit weighting based on reasoning complexity
Output: Compressed, faster, more accurate reasoning
Implementation Steps
Step 1: Prepare student and teacher conditioning
Design two prompts: one for normal reasoning and one for concise reasoning.
Sample trajectories from the current student model under normal (non-concise) prompting.
defsample_student_rollouts(model, questions, num_rollouts=4, temperature=0.7):
"""
Collect on-policy rollouts from student model.
Each question sampled multiple times to capture diversity.
"""
rollouts = []
for question in questions:
student_prompt = create_prompt_pair(question, concise=False)
for _ inrange(num_rollouts):
# Sample from student (current policy)
response = model.generate(
student_prompt,
max_tokens=256,
temperature=temperature,
top_p=0.9
)
rollouts.append({
'question': question,
'student_response': response,
'student_prompt': student_prompt
})
return rollouts
Step 3: Generate teacher responses and compute KL divergence
For each student rollout, generate concise teacher response and measure divergence.
defcompute_per_token_kl(model, student_response, teacher_response,
teacher_prompt, question, tokenizer):
"""
Compute KL divergence between teacher and student distributions.
KL(P_teacher || P_student) measures how much student diverges from teacher.
Reverse KL encourages student to match teacher (mode-seeking).
"""# Tokenize responses
student_tokens = tokenizer.encode(student_response)
teacher_tokens = tokenizer.encode(teacher_response)
# Forward passes to get logits
teacher_prompt_full = f"{teacher_prompt}\n{teacher_response}"
student_prompt_full = f"{create_prompt_pair(question, False)}\n{student_response}"
teacher_logits = model.forward(teacher_prompt_full)['logits']
student_logits = model.forward(student_prompt_full)['logits']
# Compute log probabilities for teacher and student
teacher_logprobs = softmax(teacher_logits, dim=-1)
student_logprobs = softmax(student_logits, dim=-1)
# Per-token KL divergence
per_token_kl = []
for t inrange(min(len(teacher_tokens), len(student_tokens))):
teacher_dist = teacher_logprobs[t] # Distribution over vocabulary
student_dist = student_logprobs[t]
# KL(teacher || student) = sum teacher * log(teacher / student)
kl_t = (teacher_dist * (torch.log(teacher_dist + 1e-8) -
torch.log(student_dist + 1e-8))).sum()
per_token_kl.append(kl_t.item())
return np.mean(per_token_kl)
defgenerate_teacher_responses(model, rollouts):
"""Generate concise responses from teacher for each student rollout."""for rollout in rollouts:
teacher_prompt = create_prompt_pair(rollout['question'], concise=True)
teacher_response = model.generate(
teacher_prompt,
max_tokens=128, # Shorter max for concise generation
temperature=0.5# Lower temperature for more deterministic teacher
)
rollout['teacher_response'] = teacher_response
rollout['teacher_prompt'] = teacher_prompt
# Compute KL divergence
rollout['kl_divergence'] = compute_per_token_kl(
model,
rollout['student_response'],
teacher_response,
teacher_prompt,
rollout['question'],
tokenizer
)
return rollouts
Step 4: Implicit difficulty adaptation through KL weighting
Use per-sample KL to weight loss; easy problems with large KL (big divergence) get strong compression; hard problems with small KL (teacher also verbose) get weak compression.
defcompute_difficulty_weights(rollouts, threshold=0.5):
"""
Assign difficulty weights based on KL divergence magnitude.
High KL = easy problem (teacher very concise, student verbose)
→ strong compression pressure
Low KL = hard problem (even teacher needs verbosity)
→ weak compression pressure
"""
kl_values = np.array([r['kl_divergence'] for r in rollouts])
kl_normalized = (kl_values - kl_values.min()) / (kl_values.max() - kl_values.min() + 1e-8)
weights = []
for rollout in rollouts:
kl_normalized = rollout['kl_divergence'] / (kl_values.max() + 1e-8)
# Higher KL → higher weight (compress more)
weight = kl_normalized ** 2# Quadratic to emphasize easy problems
rollout['difficulty_weight'] = weight
weights.append(weight)
return rollouts
Step 5: On-policy loss computation
Compute reverse KL divergence loss weighted by implicit difficulty.
