confu-speculative

name	confu-speculative
title	ConFu: Contemplate the Future for Better Speculative Sampling
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.08899
keywords	["Inference Optimization","Speculative Decoding","Draft Models","Token Acceptance","LLM Acceleration"]
description	Improves speculative decoding acceptance rates by exposing target model's intermediate reasoning through contemplate tokens. Achieves 8-11% acceptance rate improvement over EAGLE through future-direction guidance without extra forward passes.

ConFu: Improving Speculative Decoding Through Target Model Future Reasoning

Speculative decoding accelerates LLM inference by drafting tokens with a lightweight model and verifying with the target model. However, draft models condition only on the current prefix, causing distribution drift over multiple steps. As draft tokens diverge from target model preferences, verification acceptance rates drop, reducing speedup gains.

ConFu solves this by exposing the target model's intermediate reasoning through special contemplate tokens. These tokens trigger the target model to generate its internal predictions of future token distributions, guiding the draft model toward better candidates without requiring additional forward passes.

Core Concept

Standard speculative decoding: draft model generates token independently, target verifies

ConFu: Insert contemplate token that makes target model emit its predicted next-token distribution, share this with draft model for better guidance

The key insight: contemplate tokens allow the target model to communicate its reasoning direction without extra computation—they're generated during the verification pass and reused for subsequent draft guidance. This transforms a passive verification step into active guidance.

Architecture Overview

• Contemplate Token Mechanism: Special tokens trigger target model to encode future predictions as continuous embeddings
• Dual Prediction Path: Target model generates both next token AND contemplate embedding during verification
• Dynamic MoE Selection: Learned Mixture-of-Experts chooses context-appropriate guidance based on hidden states
• Parallel Verification: Process multiple draft candidates simultaneously using contemplate-guided predictions
• Memory Efficiency: Anchor token sampling reduces memory overhead by selective insertion

Implementation Steps

Implement contemplate token guidance in a speculative decoding framework.

Contemplate Token and Embedding Generation

import torch
import torch.nn as nn

class ContemplateMechanism(nn.Module):
    """Generates and uses contemplate embeddings for guidance."""

    def __init__(self, hidden_dim, vocab_size):
        super().__init__()
        .hidden_dim = hidden_dim
        .vocab_size = vocab_size

        
        .contemplate_token_id = vocab_size -   

        
        .guidance_encoder = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim * ),
            nn.ReLU(),
            nn.Linear(hidden_dim * , hidden_dim)
        )

        
        .moe_router = nn.Linear(hidden_dim, )  
        .moe_experts = nn.ModuleList([
            nn.Linear(hidden_dim, vocab_size)  _  ()
        ])

     ():
        
        
        guidance = .guidance_encoder(target_hidden_state)  

        
        router_logits = .moe_router(target_hidden_state)  
        router_weights = torch.softmax(router_logits, dim=)  

        
        expert_outputs = []
         expert  .moe_experts:
            expert_out = expert(target_hidden_state)  
            expert_outputs.append(expert_out)

        expert_stack = torch.stack(expert_outputs, dim=)  
        expert_logits = torch.(
            router_weights.unsqueeze() * expert_stack, dim=
        )  

         guidance, expert_logits


 (nn.Module):
    

     ():
        ().__init__()
        .base_model = base_model
        .contemplate_mechanism = ContemplateMechanism(hidden_dim, vocab_size)

     ():
        
        
        outputs = .base_model(input_ids, output_hidden_states=)
        logits = outputs.logits[:, -, :]  
        hidden_states = outputs.hidden_states[-][:, -, :]  

         return_contemplate:
            
            guidance, expert_logits = .contemplate_mechanism.get_contemplate_guidance(
                hidden_states
            )
             logits, expert_logits
        :
             logits