rollout-adaptive-supervised-finetuning-rasft

name	rollout-adaptive-supervised-finetuning-rasft
description	Rollout-Adaptive Supervised Fine-Tuning (RASFT) for reasoning tasks - policy-aware SFT that calibrates expert supervision based on problem-level solvability from verified rollouts.
authors	["Yongliang Miao","Fengyuan Liu","Wei Shi"]
date	2026-06-05T00:00:00.000Z
arxiv	2606.07006v1
tags	["reasoning","supervised-finetuning","policy-aware","adaptation","LLM"]

Rollout-Adaptive Supervised Fine-Tuning (RASFT)

Overview

Policy-aware supervised fine-tuning (SFT) that calibrates expert supervision based on verified on-policy rollouts. Avoids overfitting to single expert trajectories by incorporating correct self-generated reasoning when model already exhibits reliable behavior.

Key Innovation

Problem-Level Solvability Calibration:

• Rigid expert imitation overfits to surface forms
• Suppresses model's own reasoning distribution
• RASFT adjusts guidance strength based on model performance on each problem

Verified Rollouts:

• Estimate solvability from on-policy rollouts with verification
• Strengthen expert guidance when model struggles
• Relax imitation when model already solves correctly

Methodology

Phase 1: Solvability Estimation

For each problem p in dataset:
    # Generate multiple rollouts with current policy
    rollouts = policy.generate_n_rollouts(p, n=K)
    
    # Verify correctness of each rollout
    verified_results = verify(rollouts)
    
    # Estimate solvability: fraction of correct rollouts
    solvability(p) = count_correct(verified_results) / K

Phase 2: Adaptive Supervision

For each problem p:
    if solvability(p) is low:
        # Model struggles - strengthen expert guidance
        supervision_weight = high
        target_trajectory = expert_trajectory(p)
    elif solvability(p) is high:
        # Model reliable - relax rigid imitation
        supervision_weight = low
        # Incorporate correct self-generated trajectories
        target_trajectory = correct_self_trajectory(p)
    else:
        # Intermediate - balanced mixture
        target_trajectory = mix(expert, self_generated)

Phase 3: Policy Drift Constraint

# Clipped inverse ratio between frozen reference and current policy
reference_model = frozen_base_model
current_policy = trainable_model

drift_penalty = clip(
    inverse_ratio(
        log_prob(current_policy, trajectory),
        log_prob(reference_model, trajectory)
    ),
    min=drift_threshold
)

loss = SFT_loss + drift_penalty

Reusable Patterns

Pattern 1: Policy-Aware SFT

Use when: Standard SFT causes overfitting to expert trajectories Principle: Calibrate supervision strength based on model's own verified performance Implementation:

1. Generate rollouts with current policy
2. Verify correctness (execution check, ground truth, etc.)
3. Adjust supervision per-problem based on success rate

Pattern 2: Self-Generated Trajectory Integration

Use when: Model exhibits partial competence Approach:

• High solvability: Use model's own correct trajectories as targets
• Low solvability: Emphasize expert demonstration
• Mix for intermediate cases

Pattern 3: Reference-Based Drift Constraint

Use when: Preventing excessive deviation from base model Method:

1. Freeze base model as reference
2. Compute KL-like penalty between current and reference
3. Clip penalty to prevent over-constraint
4. Add to loss for stable adaptation

Pattern 4: Verified Rollout Sampling

Use when: Need reliable solvability estimates Steps:

1. Sample multiple rollouts per problem
2. Verify each with appropriate metric
3. Success rate → solvability score
4. Use for adaptive guidance

Implementation Considerations

Verification Methods

• Mathematical reasoning: Execute answer check
• Code reasoning: Run test cases
• General reasoning: Compare to ground truth or use validation model

Solvability Thresholds

• Low threshold (e.g., < 0.3): Full expert guidance
• High threshold (e.g., > 0.7): Self-generated focus
• Intermediate: Balanced mixture

Drift Constraint Parameters

• Clip minimum: Prevent excessive penalty
• Inverse ratio formulation: Penalize large deviations more
• Balance between adaptation and preservation

Batch-Level Processing

• Process problems in batches for efficiency
• Cache solvability estimates between epochs
• Update rollouts periodically (not every step)

Extensions

Multi-Task Adaptation

• Task-specific solvability thresholds
• Per-task expert trajectory pools

Curriculum Learning Integration

• Progress from low-solvability to high-solvability problems
• Adaptive curriculum based on rollout verification

Reinforcement Learning Hybrid

• Combine RASFT with RL fine-tuning
• Use verified rollouts as RL reward signal

Pitfalls

1. Verification Cost: Running verification for every rollout can be expensive
2. Solvability Noise: Low K (few rollouts) gives noisy estimates
3. Self-Trajectory Quality: Low-quality self trajectories degrade training
4. Drift Over-Constraint: Too tight clipping prevents necessary adaptation
5. Batch Imbalance: Solvability distribution may skew batch composition

Results (Paper Findings)

• Better than standard SFT across mathematical and code reasoning
• Outperforms SFT variants and representative RL methods
• Tested on 6 mathematical reasoning + 2 code reasoning benchmarks
• Preserves useful reasoning priors via drift constraint

Code Reference

GitHub: https://github.com/zjd1sq/RASFT

Applications

• Mathematical reasoning fine-tuning
• Code generation adaptation
• Multi-step reasoning tasks
• Tasks with verifiable outputs
• Policy-aware training for any reasoning domain

Activation Keywords

RASFT, rollout-adaptive, policy-aware SFT, verified rollouts, solvability estimation, self-generated trajectories, expert guidance calibration, reasoning fine-tuning, policy drift constraint, adaptive supervision