| name | latent-thoughts-tuning |
| title | Latent Thoughts Tuning: Bridging Context and Reasoning with Fused Information |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2602.10229 |
| keywords | ["Latent Reasoning","Chain-of-Thought","Hidden State Fusion","Model Scaling","Adaptive Reasoning Allocation"] |
| description | Enable models to reason in continuous latent space via context-prediction fusion, combining hidden state context with vocabulary embeddings to maintain scaling across model sizes. |
Latent Thoughts Tuning: Bridging Context and Reasoning with Fused Information
Problem Context
Prior latent reasoning approaches face critical limitations: directly reusing hidden states as input embeddings causes distribution mismatch and feature collapse, especially in larger models. Existing methods like Coconut degrade severely with scale (50.3% → 41.5% accuracy), while fixed reasoning schedules ignore varying problem difficulty.
Core Concept
Latent Thoughts Tuning (LT-Tuning) enables reasoning in continuous latent space through Context-Prediction Fusion—combining contextual information from hidden states with predictive semantic guidance from vocabulary embeddings. This mitigates feature collapse while enabling sample-specific reasoning diversity.
Architecture Overview
- Three-Stage Curriculum: Standard CoT fine-tuning → confidence-driven token insertion → fusion mechanism training
- Confidence-Driven Insertion: Dynamically insert "" tokens at uncertain positions based on prediction confidence
- Fusion Mechanism: Blend hidden state context with probability-weighted embeddings: e_fusion = α·h_{t-1,I} + (1-α)·e_pred
- Variable-Depth Reasoning: Adapt reasoning depth based on problem difficulty
Implementation
Phase 1: Standard CoT Fine-Tuning
def stage1_cot_finetuning(model, training_data, num_epochs=2):
"""Establish reasoning foundation with standard CoT"""
for epoch in range(num_epochs):
for example in training_data:
input_text = example['input']
reasoning = example['reasoning']
output = example['output']
full_text = f"{input_text}\n<thinking>\n{reasoning}\n</thinking>\n{output}"
logits = model.forward(full_text)
loss = cross_entropy(logits, full_text)
loss.backward()
optimizer.step()
