direct-on-policy-distillation

name	direct-on-policy-distillation
description	Weak-to-strong generalization methodology transferring RL-induced policy shifts as dense implicit reward signals from smaller to larger models, enabling cross-scale RL outcome reuse.

Direct On-Policy Distillation (Direct-OPD)

Description

A methodology for weak-to-strong generalization that transfers the RL-induced policy shift from a smaller teacher model to a stronger target model. Instead of distilling the teacher's final policy (which mixes useful RL gains with small-model limitations), Direct-OPD computes the log-ratio between post-RL and pre-RL teacher policies and uses this as a dense implicit reward for the student. Enables reusing RL outcomes across model scales without training explicit reward models or running sparse-reward RL on the target.

Activation Keywords

• Direct-OPD
• direct on-policy distillation
• weak-to-strong generalization
• RL policy transfer
• implicit reward signal
• policy shift transfer
• cross-scale RL
• 弱到强泛化
• 策略迁移
• 隐式奖励

Tools Used

• execute_code: Run RL training pipelines with distillation loss
• terminal: Execute training commands, monitor progress
• read: Load model checkpoints, compute log-ratios

Core Methodology

The Problem

• RLVR (RL with verifiable rewards) is expensive: target model must generate many rollouts during training
• Directly distilling post-RL weak teacher is insufficient: final policy mixes useful RL gains + small-model limitations
• As models scale, post-training itself becomes a bottleneck

Direct-OPD Solution

1. Run RL on a smaller model (cheap rollouts)
2. Compute log-ratio: log(π_post(x) / π_pre(x)) for each action
3. Treat this log-ratio as a dense implicit reward for the stronger student
4. Apply signal on the student's own on-policy states

Mathematical Formulation

Implicit Reward: r_impl(x, a) = log(π_weak_post(a|x) / π_weak_pre(a|x))
Student Loss: L = E[(a ~ π_strong)] [r_impl(x, a)]

The checkpoint pair (pre-RL, post-RL) tells us which actions RL made more/less likely, and Direct-OPD applies that signal on the stronger student's states.

Usage Patterns

Pattern 1: Standard Direct-OPD Training

1. Train weak model with RLVR → get π_weak_post
2. Save pre-RL checkpoint π_weak_pre
3. For student model (stronger):
   a. Generate on-policy rollouts with π_strong
   b. Compute implicit rewards: r = log(π_weak_post/π_weak_pre)
   c. Update π_strong using implicit rewards
4. Repeat until convergence

Pattern 2: Sequential Composition of Policy Shifts

Multiple weak teachers → sequential policy shifts:
1. Teacher 1 (small) → Student (medium)
2. Teacher 2 (medium, after step 1) → Student (large)
3. Each step transfers a different aspect of RL improvement

Pattern 3: Cross-Scale RL Reuse

Instead of training RL on every model size:
1. Train RL once on smallest feasible model
2. Distill policy shift to medium model
3. Distill policy shift to large model
4. Distill policy shift to very-large model

Instructions for Agents

Step 1: Prepare Teacher Model

• Train smaller model with RLVR on target task
• Save BOTH pre-RL and post-RL checkpoints
• Ensure both checkpoints use same tokenizer/vocabulary

Step 2: Compute Log-Ratio Rewards

# For each state-action pair from teacher:
log_ratio = log(π_post(a|x)) - log(π_pre(a|x))
# This is the implicit reward

Step 3: Train Student Model

# Student training loop:
for batch in student_rollouts:
    implicit_reward = compute_log_ratio(batch, teacher_post, teacher_pre)
    loss = -implicit_reward * log(π_student(a|x))
    loss.backward()
    optimizer.step()

Step 4: Validate Transfer

• Compare student performance vs:
  • Direct RL on student (baseline)
  • Standard distillation from post-RL teacher
  • Student without RL
• Measure sample efficiency improvement

Error Handling

Log-Ratio Explosion

If log-ratios become very large:

1. Clip log-ratios to reasonable range (e.g., [-5, 5])
2. Use KL-regularization to prevent student from diverging too far from reference
3. Reduce learning rate

Vocabulary Mismatch

If teacher and student have different tokenizers:

1. Map tokens to common vocabulary space
2. Use embedding-space alignment before computing log-ratios
3. Consider using subword-level probabilities

Over-Transfer

If student inherits teacher's limitations:

1. Reduce distillation strength (scale factor on implicit reward)
2. Mix implicit reward with ground-truth reward signal
3. Use multiple teachers with different strengths/weaknesses

Performance Results (from arXiv:2607.05394)

• Qwen3-1.7B: 48.3% → 62.4% on AIME 2024 (4 hours, 8 A100 GPUs)
• Outperforms step-matched direct RL
• Enables sequential composition of multiple policy shifts
• RL outcomes reusable across model scales as implicit reward signals

Related Skills

• opcod-on-policy-co-distillation — Mutual LLM improvement via co-distillation
• efficient-opd-distillation — Efficient on-policy distillation methodology
• gaussian-grpo — Gaussian Group Relative Policy Optimization
• contrastive-on-policy-thinking — CopT methodology for LLM reasoning

Resources

• Paper: arXiv:2607.05394 — "Weak-to-Strong Generalization via Direct On-Policy Distillation"
• Project: https://bytedtsinghua-sia.github.io/Direct-OPD/