latent-thoughts-tuning

name	latent-thoughts-tuning
title	Latent Thoughts Tuning: Bridging Context and Reasoning with Fused Information
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.10229
keywords	["Latent Reasoning","Chain-of-Thought","Hidden State Fusion","Model Scaling","Adaptive Reasoning Allocation"]
description	Enable models to reason in continuous latent space via context-prediction fusion, combining hidden state context with vocabulary embeddings to maintain scaling across model sizes.

Latent Thoughts Tuning: Bridging Context and Reasoning with Fused Information

Problem Context

Prior latent reasoning approaches face critical limitations: directly reusing hidden states as input embeddings causes distribution mismatch and feature collapse, especially in larger models. Existing methods like Coconut degrade severely with scale (50.3% → 41.5% accuracy), while fixed reasoning schedules ignore varying problem difficulty.

Core Concept

Latent Thoughts Tuning (LT-Tuning) enables reasoning in continuous latent space through Context-Prediction Fusion—combining contextual information from hidden states with predictive semantic guidance from vocabulary embeddings. This mitigates feature collapse while enabling sample-specific reasoning diversity.

Architecture Overview

• Three-Stage Curriculum: Standard CoT fine-tuning → confidence-driven token insertion → fusion mechanism training
• Confidence-Driven Insertion: Dynamically insert "" tokens at uncertain positions based on prediction confidence
• Fusion Mechanism: Blend hidden state context with probability-weighted embeddings: e_fusion = α·h_{t-1,I} + (1-α)·e_pred
• Variable-Depth Reasoning: Adapt reasoning depth based on problem difficulty

Implementation

Phase 1: Standard CoT Fine-Tuning

def stage1_cot_finetuning(model, training_data, num_epochs=2):
    """Establish reasoning foundation with standard CoT"""

    for epoch in range(num_epochs):
        for example in training_data:
            input_text = example['input']
            reasoning = example['reasoning']
            output = example['output']

            # Format: input + thinking + output
            full_text = f"{input_text}\n<thinking>\n{reasoning}\n</thinking>\n{output}"

            logits = model.forward(full_text)
            loss = cross_entropy(logits, full_text)
            loss.backward()
            optimizer.step()