Accelerate LLM inference using speculative decoding, Medusa multiple heads, and lookahead decoding techniques. Use when optimizing inference speed (1.5-3.6× speedup), reducing latency for real-time applications, or deploying models with limited compute. Covers draft models, tree-based attention, Jacobi iteration, parallel token generation, and production deployment strategies.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Accelerate LLM inference using speculative decoding, Medusa multiple heads, and lookahead decoding techniques. Use when optimizing inference speed (1.5-3.6× speedup), reducing latency for real-time applications, or deploying models with limited compute. Covers draft models, tree-based attention, Jacobi iteration, parallel token generation, and production deployment strategies.
classLookaheadDecoding:
def__init__(self, model, window_size=15, ngram_size=5):
self.model = model
self.W = window_size # Lookahead windowself.N = ngram_size # N-gram sizedefgenerate_step(self, tokens):
# Lookahead branch: Generate W × N candidates
candidates = {}
for w inrange(1, self.W + 1):
for n inrange(1, self.N + 1):
# Generate n-gram starting at position w
ngram = self.generate_ngram(tokens, start=w, length=n)
candidates[(w, n)] = ngram
# Verification branch: Find matching n-grams
verified = []
for ngram in candidates.values():
if ngram[0] == tokens[-1]: # First token matches last inputifself.verify(tokens, ngram):
verified.append(ngram)
# Accept longest verified n-gramreturnmax(verified, key=len) if verified else [self.model.generate_next(tokens)]
Performance:
Speedup: 1.5-2.3× (up to 3.6× for code generation)
No draft model or training needed
Works out-of-the-box with any model
Method Comparison
Method
Speedup
Training Needed
Draft Model
Quality Loss
Draft Model Speculative
1.5-2×
No
Yes (external)
None
Medusa
2-3.6×
Minimal (heads only)
No (built-in heads)
None
Lookahead
1.5-2.3×
None
No
None
Naive Batching
1.2-1.5×
No
No
None
Advanced Patterns
Training Medusa Heads
from medusa.model.medusa_model import MedusaModel
from medusa.model.kv_cache import initialize_past_key_values
import torch.nn as nn
# 1. Load base model
base_model = AutoModelForCausalLM.from_pretrained(
"lmsys/vicuna-7b-v1.3",
torch_dtype=torch.float16
)
# 2. Add Medusa heads
num_heads = 4
medusa_heads = nn.ModuleList([
nn.Linear(base_model.config.hidden_size, base_model.config.vocab_size, bias=False)
for _ inrange(num_heads)
])
# 3. Training loop (freeze base model for Medusa-1)for param in base_model.parameters():
param.requires_grad = False# Freeze base
optimizer = torch.optim.Adam(medusa_heads.parameters(), lr=1e-3)
for batch in dataloader:
# Forward pass
hidden_states = base_model(**batch, output_hidden_states=True).hidden_states[-1]
# Predict future tokens with each head
loss = 0for i, head inenumerate(medusa_heads):
logits = head(hidden_states)
# Target: tokens shifted by (i+1) positions
target = batch['input_ids'][:, i+1:]
loss += F.cross_entropy(logits[:, :-i-1], target)
# Backward
optimizer.zero_grad()
loss.backward()
optimizer.step()
Hybrid: Speculative + Medusa
# Use Medusa as draft model for speculative decoding
draft_medusa = MedusaModel.from_pretrained("medusa-vicuna-7b")
target_model = AutoModelForCausalLM.from_pretrained("vicuna-33b")
# Draft generates multiple candidates with Medusa
draft_tokens = draft_medusa.medusa_generate(prompt, max_new_tokens=5)
# Target verifies in single forward pass
outputs = target_model.generate(
prompt,
assistant_model=draft_medusa, # Use Medusa as draft
max_new_tokens=256
)
# Combines benefits: Medusa speed + large model quality
Optimal Draft Model Selection
defselect_draft_model(target_model_size, target):
"""Select optimal draft model for speculative decoding."""# Rule: Draft should be 5-10× smallerif target_model_size == "70B":
return"7B"# 10× smallerelif target_model_size == "33B":
return"7B"# 5× smallerelif target_model_size == "13B":
return"1B"# 13× smallerelse:
returnNone# Target too small, use Medusa/Lookahead instead# Example
draft = select_draft_model("70B", target_model)
# Returns "7B" → Use Llama-2-7b as draft for Llama-2-70b
Best Practices
1. Choose the Right Method
# New deployment → Medusa (best overall speedup, no draft model)if deploying_new_model:
use_method = "Medusa"# Existing deployment with small model available → Draft speculativeelif have_small_version_of_model:
use_method = "Draft Model Speculative"# Want zero training/setup → Lookaheadelif want_plug_and_play:
use_method = "Lookahead Decoding"
2. Hyperparameter Tuning
Draft Model Speculative:
# K = number of speculative tokens
K = 4# Good default
K = 2# Conservative (higher acceptance)
K = 8# Aggressive (lower acceptance, but more when accepted)# Rule: Larger K → more speedup IF draft model is good
Medusa:
# Posterior threshold (acceptance confidence)
posterior_threshold = 0.09# Standard (from paper)
posterior_threshold = 0.05# More conservative (slower, higher quality)
posterior_threshold = 0.15# More aggressive (faster, may degrade quality)# Tree depth (how many steps ahead)
medusa_choices = [[0], [0, 0], [0, 1], [0, 0, 0]] # Depth 3 (standard)
Lookahead:
# Window size W (lookahead distance)# N-gram size N (context for generation)# 7B model (more resources)
W, N = 15, 5# 13B model (moderate)
W, N = 10, 5# 33B+ model (limited resources)
W, N = 7, 5
3. Production Deployment
# vLLM with speculative decodingfrom vllm import LLM, SamplingParams
# Initialize with draft model
llm = LLM(
model="meta-llama/Llama-2-70b-hf",
speculative_model="meta-llama/Llama-2-7b-hf", # Draft model
num_speculative_tokens=5,
use_v2_block_manager=True,
)
# Generate
prompts = ["Tell me about AI:", "Explain quantum physics:"]
sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
print(output.outputs[0].text)
