relay-gen-model-switching

name	relay-gen-model-switching
title	RelayGen: Intra-Generation Model Switching for Efficient Reasoning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.06454
keywords	["Model Composition","Inference Efficiency","Reasoning Decomposition","Speculative Decoding","Cost-Quality Tradeoff"]
description	Reduce inference cost by dynamically switching from large to small LLMs during reasoning generation. Large model handles demanding reasoning phases; small model completes consolidation and answer stages triggered by discourse cues. Achieves 2.2× speedup with minimal accuracy loss.

RelayGen: Discourse-Cued Model Switching for Efficient Generation

Reasoning outputs naturally decompose into phases of varying difficulty. Early reasoning requires substantial computational power to break down problems; later consolidation and answer generation are mechanically simpler. RelayGen exploits this by switching models mid-generation: a large model handles complex reasoning, then hands off to a smaller model when discourse-level cues (Thus, Similarly, Therefore) indicate transition to easier content.

This training-free approach requires only empirically-derived switching heuristics, making it immediately applicable to any large-small model pair without retraining or fine-tuning.

Core Concept

Standard approach: use large model for entire generation, wasting compute on easy phases.

RelayGen: monitor output discourse markers for phase transitions. When the model generates cues indicating easier content (answer formatting, consolidation), switch to small model. Large model resumes for complex sections if needed.

Key insight: discourse-level cues (words like "Thus," "Therefore," "Here's the answer:") reliably signal difficulty drops, making token-level routing unnecessary.

Architecture Overview

• Large Model Path: Handles demanding reasoning steps requiring full model capacity
• Small Model Path: Completes lower-difficulty consolidation and answer phases
• Discourse Monitoring: Track output tokens for phase-transition cues
• Training-Free Switching: Use empirically-derived heuristic rules (no learned router)
• Composability: Works naturally with speculative decoding and other acceleration techniques

Implementation

Define discourse cues and switching logic:

PHASE_TRANSITION_CUES = {
    'answer': ['answer:', 'solution:', 'result:', 'therefore,', 'thus,', 'finally,'],
    'consolidation': ['in conclusion', 'to summarize', 'simplifying', 'therefore'],
    'calculation': ['{', '}', '=']
}

def ():
    
    lower_text = generated_text.lower()

    
     cue  PHASE_TRANSITION_CUES[]:
         cue  lower_text:
             , 

    
     cue  PHASE_TRANSITION_CUES[]:
         cue  lower_text:
             , 

     , current_phase

 ():
    
     generated_text[-window_size:]

 :
    

     ():
        .large_model = large_model
        .small_model = small_model
        .switch_threshold = switch_threshold
        .max_tokens = max_tokens

     ():
        
        generated = 
        current_model = 
        tokens_generated = 

         tokens_generated < .max_tokens:
            
            model = .large_model  current_model ==   .small_model

            
            next_token = model.generate_token(
                prompt + generated,
                temperature=temperature
            )

             next_token == :  
                

            generated += next_token
            tokens_generated += 

            
             tokens_generated %  == :
                recent = get_last_token_window(generated, window_size=)
                should_switch, new_phase = should_switch_to_small_model(
                    recent,
                    current_phase=  current_model ==   
                )

                 should_switch  current_model == :
                    ()
                    current_model = 

         generated

     ():
        
        generated = 
        current_model = 
        tokens_generated = 

         tokens_generated < .max_tokens:
             current_model == :
                model = .large_model
            :
                
                model = .small_model

            
            speculative_tokens = model.generate_tokens(
                prompt + generated,
                num_tokens=speculation_length  current_model ==   ,
                temperature=temperature
            )

             spec_token  speculative_tokens:
                generated += spec_token
                tokens_generated += 

                
                 tokens_generated %  == :
                    recent = get_last_token_window(generated, window_size=)
                    should_switch, new_phase = should_switch_to_small_model(recent)

                     should_switch  current_model == :
                        current_model = 
                          

                 spec_token == :
                     generated

         generated