adaptive-speculative-decoding

name	adaptive-speculative-decoding
title	OmniDraft: A Cross-vocabulary, Online Adaptive Drafter for On-device Speculative Decoding
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.02659
keywords	["Speculative Decoding","On-device Inference","Online Adaptation","Vocabulary Mismatch","Knowledge Distillation"]
description	Accelerate LLM inference 1.5-2x using a universal draft model that adapts to user data in real-time, handling different target models and tokenizers via online n-gram cache and hybrid distillation.

OmniDraft: Universal and Adaptive Speculative Decoding

Speculative decoding accelerates LLM inference by using a small draft model to generate multiple tokens, which are verified by the target model in parallel. However, deploying this approach on-device faces challenges: (1) draft models trained for one target model may not work with another, and (2) user-specific data causes distribution shift over time. OmniDraft addresses both through a single universal draft model (Llama-68M) that works with any target model via cross-vocabulary token mapping, and online knowledge distillation that continuously adapts the drafter to user data without explicit retraining.

The key innovations are an online n-gram cache that translates between different tokenizer vocabularies, and a hybrid loss combining token-level and distribution-level distillation that updates the drafter as it processes user data. This enables 1.5-2x speedup across reasoning, coding, and text generation tasks on consumer hardware.

Core Concept

OmniDraft operates on two principles: (1) vocabulary-aware adaptation through n-gram caches that learn token mappings between drafters and targets, and (2) online learning that improves the drafter continuously as it encounters user data. Rather than training separate drafters for each target model or retraining when deployment contexts change, a single universal drafter becomes increasingly specialized to its deployment context through lightweight online updates.

The n-gram cache maintains learned mappings showing which draft tokens typically correspond to which target tokens, enabling effective translation even with completely different vocabularies. Online distillation ensures the drafter learns patterns specific to the user's domain without catastrophic forgetting of general knowledge.

Architecture Overview

The system comprises several interconnected components:

• Universal Draft Model: Lightweight Llama-68M serving as single drafter for multiple targets
• N-gram Cache: Learned bijection between draft and target token sequences, enabling vocabulary translation
• Hybrid Distillation Head: Combines KL divergence (for directly mapped tokens) and NLL (for n-gram mapped tokens)
• Acceptance Predictor: Lightweight head predicting token acceptance probability, enabling adaptive proposal lengths
• Online Learning Loop: Continuously updates draft model and cache during inference with user data

Implementation

Start with the vocabulary mapping layer:

import torch
import torch.nn as nn
from typing  , , 
 collections  defaultdict

 :
    

     ():
        .n = n
        .cache_size = cache_size

        
        .cache = defaultdict(: defaultdict())
        .total_accesses = 

     ():
        
        
         (draft_sequence) >= .n:
            key = (draft_sequence[-(.n-):])
        :
            key = (draft_sequence)

        .cache[key][target_token] += 
        .total_accesses += 

        
         (.cache) > .cache_size:
            ._evict_cold_entries()

     () -> [[, ]]:
        
         (draft_sequence) >= .n:
            key = (draft_sequence[-(.n-):])
        :
            key = (draft_sequence)

         key   .cache:
            
             []

        
        token_counts = .cache[key]
        total = (token_counts.values())

        predictions = [
            (token_id, count / total)
             token_id, count  (
                token_counts.items(),
                key= x: x[],
                reverse=
            )[:top_k]
        ]

         predictions

     ():
        
        
        frequencies = {}
         key, token_dict  .cache.items():
            frequencies[key] = (token_dict.values())

        
        num_to_remove = (.cache) // 
        to_remove = (
            frequencies.items(),
            key= x: x[]
        )[:num_to_remove]

         key, _  to_remove:
             .cache[key]

 (nn.Module):
    

     ():
        ().__init__()
        .draft_vocab_size = draft_vocab_size
        .target_vocab_size = target_vocab_size

        
        .adaptation_proj = nn.Linear(hidden_dim, hidden_dim)

        
        .ngram_cache = NGramCache(n=)

     () -> torch.Tensor:
        
        batch_size, seq_len, _ = draft_logits.shape

        
        target_logits = torch.full(
            (batch_size, seq_len, .target_vocab_size),
            (),
            device=draft_logits.device
        )

        
         pos  (seq_len):
            
            draft_pos_logits = draft_logits[:, pos, :]

            
            mappings = .ngram_cache.get_mapping(draft_history, top_k=)

             target_token_id, mapping_prob  mappings:
                 target_token_id < .target_vocab_size:
                    
                     batch_idx  (batch_size):
                        
                        current_val = target_logits[batch_idx, pos, target_token_id]
                        boost = torch.log(torch.tensor(mapping_prob + ))
                        target_logits[batch_idx, pos, target_token_id] = (
                            current_val, boost
                        )

         target_logits