semcot-accelerated-cot-implicit-tokens

name	semcot-accelerated-cot-implicit-tokens
title	SemCoT: Accelerating Chain-of-Thought via Semantically-Aligned Implicit Tokens
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.24940
keywords	["Chain-of-Thought","Knowledge Distillation","Semantic Alignment","Inference Optimization","Implicit Reasoning"]
description	Encode reasoning steps as hidden embeddings instead of explicit text using contrastively-trained sentence transformers and lightweight distilled models, reducing token generation cost while preserving semantic alignment with ground-truth reasoning.

Title: Compress Reasoning Into Hidden Embeddings With Semantic Fidelity

Chain-of-thought reasoning improves accuracy but costs extra tokens. SemCoT compresses explicit reasoning steps into implicit tokens—single vectors in the LLM's hidden space—while maintaining semantic fidelity through contrastive alignment. A sentence transformer trained on reasoning pairs guides a lightweight student model to generate semantically-equivalent implicit tokens without explicit textual generation.

The approach combines two key ideas: (1) contrastive alignment ensures implicit representations preserve ground-truth reasoning semantics, and (2) knowledge distillation from a student model reduces per-token cost.

Core Concept

Implicit Chain-of-Thought with Semantic Preservation:

• Explicit CoT: Generate full reasoning steps as text tokens (slow, verbose)
• SemCoT: Generate special <CoT> tokens whose embeddings encode reasoning (fast, compressed)
• Contrastive Training: Align implicit embeddings with explicit reasoning via learned metric
• Lightweight Generator: Small distilled model produces implicit tokens efficiently
• Dual Loss: Accuracy loss (correct answers) + semantic alignment loss (reasoning fidelity)

Architecture Overview

• Sentence Transformer: Trained to measure semantic similarity between reasoning pairs (uses LLM middle layers)
• Implicit Token Generator: Lightweight language model (distilled/pruned from target LLM)
• Embedding Projection: Linear transformation from student embedding space to main LLM space
• Dual Optimization: Cross-entropy loss + semantic alignment via contrastive similarity
• Inference Mode: Single pass generating implicit <CoT> tokens for each query

Implementation Steps

1. Train Contrastive Sentence Transformer

Create a metric that measures whether implicit and explicit reasoning are semantically equivalent.

class ReasoningSentenceTransformer(nn.Module):
    def __init__(self, llm, hidden_dim=768):
        self.llm = llm
        # Use middle 5 layers of LLM as backbone
        .backbone = llm.transformer.h[:]  
        .pooling = nn.AdaptiveAvgPool1d()
        .projection = nn.Linear(hidden_dim, )

     ():
        
        hidden = .backbone(input_ids, attention_mask)[-]
        
        pooled = .pooling(hidden.transpose(, )).squeeze(-)
        
        embeddings = .projection(pooled)
         embeddings  

 ():
    optimizer = torch.optim.Adam(transformer.parameters(), lr=)

     epoch  (epochs):
         batch  reasoning_pairs:
            explicit_reasoning = batch[]  
            condensed_reasoning = batch[]  

            
            emb_explicit = transformer(explicit_reasoning)
            emb_condensed = transformer(condensed_reasoning)

            
            pos_sim = F.cosine_similarity(emb_explicit, emb_condensed)
            pos_loss =  - pos_sim.mean()

            
            shuffled_indices = torch.randperm((batch))
            emb_negatives = emb_explicit[shuffled_indices]
            neg_sim = F.cosine_similarity(emb_explicit, emb_negatives)
            neg_loss = torch.clamp(neg_sim + , =).mean()

            loss = pos_loss + neg_loss
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

     transformer