Accelerate LLM inference using speculative decoding, Medusa multiple heads, and lookahead decoding techniques. Use when optimizing inference speed (1.5-3.6× speedup), reducing latency for real-time applications, or deploying models with limited compute. Covers draft models, tree-based attention, Jacobi iteration, parallel token generation, and production deployment strategies.
Accelerate LLM inference using speculative decoding, Medusa multiple heads, and lookahead decoding techniques. Use when optimizing inference speed (1.5-3.6× speedup), reducing latency for real-time applications, or deploying models with limited compute. Covers draft models, tree-based attention, Jacobi iteration, parallel token generation, and production deployment strategies.
classLookaheadDecoding:
def__init__(self, model, window_size=15, ngram_size=5):
self.model = model
self.W = window_size # Lookahead windowself.N = ngram_size # N-gram sizedefgenerate_step(self, tokens):
# Lookahead branch: Generate W × N candidates
candidates = {}
for w inrange(1, self.W + 1):
for n inrange(1, self.N + 1):
# Generate n-gram starting at position w
ngram = self.generate_ngram(tokens, start=w, length=n)
candidates[(w, n)] = ngram
# Verification branch: Find matching n-grams
verified = []
for ngram in candidates.values():
if ngram[0] == tokens[-1]: # First token matches last inputifself.verify(tokens, ngram):
verified.append(ngram)
# Accept longest verified n-gramreturnmax(verified, key=len) if verified else [self.model.generate_next(tokens)]
Performance:
Speedup: 1.5-2.3× (up to 3.6× for code generation)
No draft model or training needed
Works out-of-the-box with any model
Method Comparison
Method
Speedup
Training Needed
Draft Model
Quality Loss
Draft Model Speculative
1.5-2×
No
Yes (external)
None
Medusa
2-3.6×
Minimal (heads only)
No (built-in heads)
None
Lookahead
1.5-2.3×
None
No
None
Naive Batching
1.2-1.5×
No
No
None
Advanced Patterns
Training Medusa Heads
from medusa.model.medusa_model import MedusaModel
from medusa.model.kv_cache import initialize_past_key_values
import torch.nn as nn
# 1. Load base model
base_model = AutoModelForCausalLM.from_pretrained(
"lmsys/vicuna-7b-v1.3",
torch_dtype=torch.float16
)
# 2. Add Medusa heads
num_heads = 4
medusa_heads = nn.ModuleList([
nn.Linear(base_model.config.hidden_size, base_model.config.vocab_size, bias=False)
for _ inrange(num_heads)
])
# 3. Training loop (freeze base model for Medusa-1)for param in base_model.parameters():
param.requires_grad = False# Freeze base
optimizer = torch.optim.Adam(medusa_heads.parameters(), lr=1e-3)
for batch in dataloader:
# Forward pass
hidden_states = base_model(**batch, output_hidden_states=True).hidden_states[-1]
# Predict future tokens with each head
loss = 0for i, head inenumerate(medusa_heads):
logits = head(hidden_states)
# Target: tokens shifted by (i+1) positions
target = batch['input_ids'][:, i+1:]
loss += F.cross_entropy(logits[:, :-i-1], target)
# Backward
optimizer.zero_grad()
loss.backward()
optimizer.step()
Hybrid: Speculative + Medusa
# Use Medusa as draft model for speculative decoding
draft_medusa = MedusaModel.from_pretrained("medusa-vicuna-7b")
target_model = AutoModelForCausalLM.from_pretrained("vicuna-33b")
# Draft generates multiple candidates with Medusa
draft_tokens = draft_medusa.medusa_generate(prompt, max_new_tokens=5)
# Target verifies in single forward pass
outputs = target_model.generate(
prompt,
assistant_model=draft_medusa, # Use Medusa as draft
max_new_tokens=256
)
# Combines benefits: Medusa speed + large model quality
Optimal Draft Model Selection
defselect_draft_model(target_model_size, target):
"""Select optimal draft model for speculative decoding."""# Rule: Draft should be 5-10× smallerif target_model_size == "70B":
return"7B"# 10× smallerelif target_model_size == "33B":
return"7B"# 5× smallerelif target_model_size == "13B":
return"1B"# 13× smallerelse:
returnNone# Target too small, use Medusa/Lookahead instead# Example
draft = select_draft_model("70B", target_model)
# Returns "7B" → Use Llama-2-7b as draft for Llama-2-70b
Best Practices
1. Choose the Right Method
# New deployment → Medusa (best overall speedup, no draft model)if deploying_new_model:
use_method = "Medusa"# Existing deployment with small model available → Draft speculativeelif have_small_version_of_model:
use_method = "Draft Model Speculative"# Want zero training/setup → Lookaheadelif want_plug_and_play:
use_method = "Lookahead Decoding"
2. Hyperparameter Tuning
Draft Model Speculative:
# K = number of speculative tokens
K = 4# Good default
K = 2# Conservative (higher acceptance)
K = 8# Aggressive (lower acceptance, but more when accepted)# Rule: Larger K → more speedup IF draft model is good
Medusa:
# Posterior threshold (acceptance confidence)
posterior_threshold = 0.09# Standard (from paper)
posterior_threshold = 0.05# More conservative (slower, higher quality)
posterior_threshold = 0.15# More aggressive (faster, may degrade quality)# Tree depth (how many steps ahead)
medusa_choices = [[0], [0, 0], [0, 1], [0, 0, 0]] # Depth 3 (standard)
Lookahead:
# Window size W (lookahead distance)# N-gram size N (context for generation)# 7B model (more resources)
W, N = 15, 5# 13B model (moderate)
W, N = 10, 5# 33B+ model (limited resources)
W, N = 7, 5
3. Production Deployment
# vLLM with speculative decodingfrom vllm import LLM, SamplingParams
# Initialize with draft model
llm = LLM(
model="meta-llama/Llama-2-70b-hf",
speculative_model="meta-llama/Llama-2-7b-hf", # Draft model
num_speculative_tokens=5,
use_v2_block_manager=True,
)
# Generate
prompts = ["Tell me about AI:", "Explain quantum physics:"]
sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
print(output.outputs[0].text)
