| name | confu-speculative |
| title | ConFu: Contemplate the Future for Better Speculative Sampling |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2603.08899 |
| keywords | ["Inference Optimization","Speculative Decoding","Draft Models","Token Acceptance","LLM Acceleration"] |
| description | Improves speculative decoding acceptance rates by exposing target model's intermediate reasoning through contemplate tokens. Achieves 8-11% acceptance rate improvement over EAGLE through future-direction guidance without extra forward passes. |
ConFu: Improving Speculative Decoding Through Target Model Future Reasoning
Speculative decoding accelerates LLM inference by drafting tokens with a lightweight model and verifying with the target model. However, draft models condition only on the current prefix, causing distribution drift over multiple steps. As draft tokens diverge from target model preferences, verification acceptance rates drop, reducing speedup gains.
ConFu solves this by exposing the target model's intermediate reasoning through special contemplate tokens. These tokens trigger the target model to generate its internal predictions of future token distributions, guiding the draft model toward better candidates without requiring additional forward passes.
Core Concept
Standard speculative decoding: draft model generates token independently, target verifies
ConFu: Insert contemplate token that makes target model emit its predicted next-token distribution, share this with draft model for better guidance
The key insight: contemplate tokens allow the target model to communicate its reasoning direction without extra computation—they're generated during the verification pass and reused for subsequent draft guidance. This transforms a passive verification step into active guidance.
Architecture Overview
- Contemplate Token Mechanism: Special tokens trigger target model to encode future predictions as continuous embeddings
- Dual Prediction Path: Target model generates both next token AND contemplate embedding during verification
- Dynamic MoE Selection: Learned Mixture-of-Experts chooses context-appropriate guidance based on hidden states
- Parallel Verification: Process multiple draft candidates simultaneously using contemplate-guided predictions
- Memory Efficiency: Anchor token sampling reduces memory overhead by selective insertion
Implementation Steps
Implement contemplate token guidance in a speculative decoding framework.
Contemplate Token and Embedding Generation
import torch
import torch.nn as nn
class ContemplateMechanism(nn.Module):
"""Generates and uses contemplate embeddings for guidance."""
def __init__(self, hidden_dim, vocab_size):
super().__init__()
.hidden_dim = hidden_dim
.vocab_size = vocab_size
.contemplate_token_id = vocab_size -
.guidance_encoder = nn.Sequential(
nn.Linear(hidden_dim, hidden_dim * ),
nn.ReLU(),
nn.Linear(hidden_dim * , hidden_dim)
)
.moe_router = nn.Linear(hidden_dim, )
.moe_experts = nn.ModuleList([
nn.Linear(hidden_dim, vocab_size) _ ()
])
():
guidance = .guidance_encoder(target_hidden_state)
router_logits = .moe_router(target_hidden_state)
router_weights = torch.softmax(router_logits, dim=)
expert_outputs = []
expert .moe_experts:
expert_out = expert(target_hidden_state)
expert_outputs.append(expert_out)
expert_stack = torch.stack(expert_outputs, dim=)
expert_logits = torch.(
router_weights.unsqueeze() * expert_stack, dim=
)
guidance, expert_logits
(nn.Module):
():
().__init__()
.base_model = base_model
.contemplate_mechanism = ContemplateMechanism(hidden_dim, vocab_size)
():
outputs = .base_model(input_ids, output_hidden_states=)
logits = outputs.logits[:, -, :]
hidden_states = outputs.hidden_states[-][:, -, :]
return_contemplate:
guidance, expert_logits = .contemplate_mechanism.get_contemplate_guidance(
hidden_states
)
logits, expert_logits
:
logits
