pretraining-data-refinement

name	pretraining-data-refinement
title	RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programs
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.03253
keywords	["Data Refinement","Pretraining Efficiency","Distillation","Program Extraction","Text Cleaning"]
description	Improve pretraining efficiency by refining noisy data through expert-guided programs: learn to generate deletion operations that clean documents, achieving 2.6-7.2% performance gains with fewer training tokens.

RefineX: Scalable Data Refinement via Learned Edit Programs

Pretraining data quality directly impacts model performance, but refining large-scale corpora is expensive. RefineX proposes learning what to delete: an expert LLM generates refined versions of documents, the system extracts minimal edit operations (deletions only), and a compact model learns to apply those operations at scale. This approach achieves 2.6%-7.2% improvements on downstream tasks using significantly fewer training tokens, with the advantage that edit programs are interpretable and don't introduce hallucinated content.

The key insight is that refinement doesn't require complex edits—deletion suffices for many quality issues. Removing duplicates, irrelevant sections, noisy content, and low-quality text often improves downstream performance more than inserting or rewriting. By constraining edits to deletions and learning programmatic operations, RefineX avoids the hallucination risks of generative refinement while remaining computationally efficient.

Core Concept

RefineX operates in three phases. First, an expert LLM (Qwen2.5-72B) refines documents by removing low-quality content, emphasizing deletions over rewrites. Second, the system identifies minimal deletion operations needed to transform original text into refined versions using edit distance algorithms. Third, a compact student model (0.6B parameters) learns to generate three functions—remove_lines(), remove_str(), and keep_all()—which execute efficiently. The student model applies these operations to the full pretraining corpus at scale.

This three-stage approach balances quality (expert refinement), efficiency (compact student model), and interpretability (explicit deletion operations). The result is cleaner pretraining data with measurable improvements across multiple downstream tasks.

Architecture Overview

The system comprises three stages:

• End-to-End Refinement: Expert LLM generates cleaned versions emphasizing deletion-based operations, producing high-quality supervision
• Program Extraction: Minimum edit distance algorithms identify minimal deletion operations transforming original to refined text, discarding insertions/replacements
• Model Distillation & Execution: Compact 0.6B model learns to predict removal operations, which execute as simple string manipulations at scale

Implementation

Start with the program extraction engine using edit distance:

from difflib import SequenceMatcher
from typing import List, , 
 re

 :
    

     ():
        .operations = []

     () -> [[, ]]:
        
        original_lines = original.split()
        refined_lines = refined.split()

        
        matcher = SequenceMatcher(, original_lines, refined_lines)
        matching_blocks = matcher.get_matching_blocks()

        
        deletions = []
        last_orig_end = 

         block  matching_blocks:
            orig_start, refined_start, size = block.a, block.b, block.size

            
             orig_start > last_orig_end:
                deletions.append((last_orig_end, orig_start))

            last_orig_end = orig_start + size

        
         last_orig_end < (original_lines):
            deletions.append((last_orig_end, (original_lines)))

         deletions

     () -> []:
        
        deletions = []

        
        matcher = SequenceMatcher(, original, refined)
        matching_blocks = matcher.get_matching_blocks()

        last_orig_end = 
         block  matching_blocks:
            orig_start, refined_start, size = block.a, block.b, block.size

            
             orig_start > last_orig_end:
                deleted_text = original[last_orig_end:orig_start]
                 (deleted_text) <= max_span_length:
                    deletions.append({
                        : last_orig_end,
                        : orig_start,
                        : deleted_text
                    })

            last_orig_end = orig_start + size

        
         last_orig_end < (original):
            deleted_text = original[last_orig_end:]
             (deleted_text) <= max_span_length:
                deletions.append({
                    : last_orig_end,
                    : (original),
                    : deleted_text
                })

         deletions

     () -> :
        
        line_deletions = .extract_line_deletions(original, refined)
        substring_deletions = .extract_substring_deletions(original, refined)

        
         (line_deletions) <= :
            
            operations = [{
                : ,
                : line_deletions
            }]
        :
            
            operations = [{
                : ,
                : substring_deletions
            }]

         {
            : (original),
            : (refined),
            : (refined) / ((original) + ),
            : operations
        }