arbitrage-advantage-speculation

name	arbitrage-advantage-speculation
title	Arbitrage: Efficient Reasoning via Advantage-Aware Speculation
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.05033
keywords	["speculative decoding","reasoning efficiency","dynamic routing","draft models","semantic verification"]
description	Route generation dynamically based on relative model advantage for 2× latency reduction in reasoning. Arbitrage learns when draft models excel versus when target models are worthwhile—critical for balancing cost and quality in long reasoning chains.

Overview

Arbitrage improves speculative decoding by introducing a lightweight router trained to identify when the target model will produce meaningfully superior reasoning steps. Rather than fixed acceptance thresholds, the framework dynamically routes generation, achieving near-optimal efficiency-accuracy tradeoffs.

When to Use

• Reasoning tasks with lengthy chain-of-thought processes
• Inference cost reduction without quality loss
• Scenarios with heterogeneous model capabilities
• Mathematical and logical reasoning problems
• Need for dynamic quality-efficiency tradeoff

When NOT to Use

• Simple single-step generation
• Strictly real-time latency-critical applications
• Cases where draft models are unavailable
• Scenarios where all steps need identical quality

Core Technique

Dynamic routing via learned advantage estimation:

# Arbitrage: Dynamic routing based on model advantage
class ArbitrageRouter:
    def __init__(self, draft_model, target_model):
        self.draft = draft_model
        self.target = target_model
        # Lightweight router predicting when target exceeds draft
        self.router = nn.Sequential(
            nn.Linear(hidden_dim, 256),
            nn.ReLU(),
            nn.Linear(256, 1),
            nn.Sigmoid()
        )

    def predict_target_advantage(self, state):
        """Router predicts probability target model is superior."""
        # Extract features from current state
        features = self.extract_state_features(state)
        # Predict target advantage
        prob_target_better = self.router(features)
        return prob_target_better

    def generate_with_dynamic_routing():
        
        state = prompt
        total_steps = 

         step  (num_steps):
            
            target_advantage = .predict_target_advantage(state)

             target_advantage > :
                
                next_step = .target.generate_step(state)
            :
                
                next_step = .draft.generate_step(state)

            state = state + next_step
            total_steps += 

         state

     ():
        
         trajectory  trajectories:
             step_idx, (state, action)  (trajectory):
                
                draft_output = .draft.generate_step(state)
                target_output = .target.generate_step(state)

                
                advantage = .compute_semantic_advantage(
                    draft_output,
                    target_output,
                    trajectory[step_idx+:]  
                )

                
                features = .extract_state_features(state)
                pred_advantage = .router(features)

                loss = torch.nn.functional.mse_loss(
                    pred_advantage,
                    torch.tensor([advantage])
                )
                loss.backward()

        .optimizer.step()