mirror-speculative-decoding

name	mirror-speculative-decoding
title	Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.13161
keywords	["inference-optimization","speculative-decoding","parallel-execution","heterogeneous-hardware"]
description	Run draft and target models in parallel across GPU/NPU using bidirectional speculation: draft predicts forward continuations while target speculates correction paths simultaneously. Achieve 2.8x-5.8x wall-time speedup on 14B-66B models.

Mirror Speculative Decoding: Parallel Heterogeneous Inference Acceleration

Standard speculative decoding runs draft model serially then validates with target model, creating latency bottleneck. Mirror Speculative Decoding launches both draft and target speculatively in parallel on heterogeneous hardware (GPU + NPU), with draft speculating forward and target speculating corrections simultaneously.

Core insight: traditional serial pipelines waste hardware. By running bidirectional speculation concurrently on different devices, you break the serial barrier while maintaining high token acceptance rates, achieving 2.8-5.8x speedup.

Core Concept

Bidirectional Speculation: Draft model predicts forward token sequences while target model speculatively predicts correction/alternative paths. Both run in parallel.

Heterogeneous Hardware Exploitation: Leverage GPU and NPU concurrency—draft model on faster device, target on secondary device, minimizing idle time.

Multi-Token Streaming: Draft emits multiple tokens per step without sacrificing verification quality.

Architecture Overview

• Draft Model Pipeline: Generates candidate sequences on primary device
• Target Model Pipeline: Speculatively generates corrections on secondary device
• Token Synchronization: Merge paths when they diverge
• Acceptance Logic: Verify candidate tokens match target predictions

Implementation Steps

Stage 1: Setup Heterogeneous Execution

Configure dual-device inference:

import torch
import torch.nn as nn
from transformers import AutoModelForCausalLM, AutoTokenizer

class HeterogeneousModelPair:
    def __init__(self, model_name, draft_device='cuda:0', target_device='cuda:1'):
        """
        Load draft and target models on different devices.
        """

        self.tokenizer = AutoTokenizer.from_pretrained(model_name)

        # Draft model: smaller, faster
        self.draft_model = AutoModelForCausalLM.from_pretrained(
            ,  
            torch_dtype=torch.float16
        ).to(draft_device)

        
        .target_model = AutoModelForCausalLM.from_pretrained(
            model_name,
            torch_dtype=torch.float16
        ).to(target_device)

        .draft_device = draft_device
        .target_device = target_device

        
         param  .draft_model.parameters():
            param.requires_grad = 
         param  .target_model.parameters():
            param.requires_grad = 

     ():
        

         torch.no_grad():
            
            draft_ids = input_ids.to(.draft_device)

            draft_tokens = []
            draft_logprobs = []

             _  (num_speculative_tokens):
                logits = .draft_model(draft_ids).logits[:, -, :]

                
                log_probs = torch.nn.functional.log_softmax(logits, dim=-)

                
                next_token = torch.multinomial(
                    torch.exp(log_probs),
                    num_samples=
                )

                draft_tokens.append(next_token)
                draft_logprobs.append(log_probs[, next_token].item())

                
                draft_ids = torch.cat([draft_ids, next_token], dim=-)

        draft_sequence = torch.cat(draft_tokens, dim=-)

         draft_sequence, draft_logprobs

     ():
        

         torch.no_grad():
            target_ids = input_ids.to(.target_device)

            correction_tokens = []
            correction_logprobs = []

             _  (num_correction_tokens):
                logits = .target_model(target_ids).logits[:, -, :]

                
                log_probs = torch.nn.functional.log_softmax(logits, dim=-)

                
                next_token = torch.multinomial(
                    torch.exp(log_probs),
                    num_samples=
                )

                correction_tokens.append(next_token)
                correction_logprobs.append(log_probs[, next_token].item())

                target_ids = torch.cat([target_ids, next_token], dim=-)

        correction_sequence = torch.cat(correction_tokens, dim=-)

         correction_sequence, correction_logprobs