surgical-post-training-error-correction

name	surgical-post-training-error-correction
title	Surgical Post-Training: Cutting Errors, Keeping Knowledge
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.01683
keywords	["Post-Training","Error Correction","Data Rectification","Binary Classification","Reasoning Alignment"]
description	Correct reasoning errors with minimal data collection by using an oracle to surgically fix only erroneous steps in existing trajectories. Use binary classification loss on rectified pairs with implicit KL regularization to prevent knowledge forgetting.

Surgical Post-Training: Surgical Error Correction with Knowledge Preservation

Standard post-training data collection requires extensive high-quality annotation. Surgical Post-Training (SPoT) takes a different approach: starting with existing model-generated trajectories, use a stronger oracle model to surgically correct only the erroneous reasoning steps while preserving the original trajectory structure and style. This minimal-edit approach dramatically reduces annotation burden while maintaining knowledge from pretraining.

The core insight is that error correction should be surgical, not wholesale. Most of a trajectory is correct; fixing only the broken parts minimizes distribution shift. Combined with binary classification objectives and implicit KL regularization through reward-based loss, SPoT achieves consistent improvements with 4,000-10,000 rectified pairs instead of the 100,000+ examples typical approaches require.

Core Concept

SPoT operates through three coordinated mechanisms:

1. Data Rectification Pipeline: Generate erroneous responses from the model, use an oracle to fix only the steps that fail, keep everything else unchanged
2. Trajectory Similarity Enforcement: Filter pairs to ensure they differ only minimally (>60% token overlap), preventing major distribution shifts
3. Binary Classification Objective: Treat correct vs. incorrect reasoning as a classification problem, then regularize via a KL constraint acting as an elastic tether to prevent catastrophic forgetting

Architecture Overview

• Input: Erroneous trajectories from base model, oracle model for correction
• Rectification: Oracle identifies step-level errors, proposes minimal fixes
• Similarity Filtering: Keep only pairs with high token-overlap ratio (LCS-based)
• Loss Design: Binary classification loss with KL-based regularization
• Output: Fine-tuned model with improved reasoning accuracy

Implementation Steps

Step 1: Generate candidate erroneous and correct pairs

Sample trajectories from the model, identify failures, and use oracle for correction.

def generate_rectified_pairs(base_model, oracle_model, prompt_batch,
                            num_samples_per_prompt=2):
    """Generate erroneous model outputs and oracle corrections."""
    rectified_pairs = []

     prompt  prompt_batch:
        
        candidates = [base_model.generate(prompt)  _  (num_samples_per_prompt)]

         candidate  candidates:
            
            is_correct, error_steps = oracle_model.evaluate(candidate)

              is_correct  (error_steps) > :
                
                corrected = oracle_model.fix_steps(candidate, error_steps)

                rectified_pairs.append({
                    : candidate,
                    : corrected,
                    : error_steps
                })

     rectified_pairs