compress-distill-reasoning-trace-compression

name	compress-distill-reasoning-trace-compression
description	Post-hoc compression of reasoning chain-of-thought traces before knowledge distillation for efficient training and inference
version	1.0.0
category	ai_collection
tags	["deep-learning","distillation","efficiency","reasoning","knowledge-transfer"]
arxiv	2606.05988v1
paper_title	Compress-Distill: Reasoning Trace Compression for Efficient Knowledge Distillation
authors	["Maxime Griot","Paul Steven Scotti","Tanishq Mathew Abraham"]
published	2026-06-04T00:00:00.000Z
activation_keywords	["reasoning distillation","trace compression","knowledge distillation","CoT compression","efficient training","student model"]

Compress-Distill: Reasoning Trace Compression

Core Insight

Reasoning trace compression offers accuracy-efficiency trade-off: students retain 96% accuracy while gaining 18x higher per-token efficiency.

Methodology

Compression Pipeline

1. Generate teacher traces: Qwen3.5-397B-A17B, gpt-oss-120B produce ~283k traces
2. Compress with instruction-tuned models: reduce to 8.6-21.0% of original length
3. Distill compressed traces: train student models on compressed reasoning

Key Findings

• Training speed: 2.0-7.6x faster with compressed traces
• Token efficiency: 12-30% of raw training tokens
• Inference output length: 3-19x shorter
• Accuracy: compressed retains up to 96% of raw-trace performance

Trade-off Analysis

Raw traces → Highest accuracy at every scale
Compressed → Higher efficiency with accuracy trade-off
Truncated → Lower accuracy than model-compressed

Critical insight: Compression ≠ mere truncation

• Model-compressed beats naive truncation (especially for smaller students)
• Maintains shorter inference outputs while preserving reasoning quality

Implementation Pattern

Compression Strategy

# Conceptual compression pipeline
def compress_reasoning_trace(teacher_trace, compressor_model):
    # Step 1: Extract key reasoning steps
    semantic_steps = extract_semantic_operations(teacher_trace)
    
    # Step 2: Compress while preserving logic
    compressed = compressor_model.compress(
        trace=teacher_trace,
        preserve_logic=True,
        target_ratio=0.15  # 15% of original length
    )
    
    # Step 3: Validate compressed reasoning
    if validate_reasoning_chain(compressed):
        return compressed
    
def distill_with_compression(student, compressed_traces):
    
     student.train(compressed_traces, epochs=...)