relayllm-efficient-reasoning-collaboration

name	relayllm-efficient-reasoning-collaboration
title	RelayLLM: Efficient Reasoning via Collaborative Decoding
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2601.05167
keywords	["Efficient Inference","Multi-Model Collaboration","Token-Level Routing","Reasoning Optimization"]
description	Enable small language models to dynamically invoke larger models at critical reasoning tokens rather than offloading entire queries. RelayLLM achieves 49.52% accuracy across benchmarks while invoking the large model for only 1.07% of tokens—98.2% cost reduction compared to non-collaborative approaches.

When to Use This Skill

• Reasoning tasks where a small model can handle most steps independently
• Cost-sensitive inference with access to both capable and efficient models
• Applications requiring balanced accuracy-efficiency trade-offs
• Multi-benchmark scenarios (mathematical reasoning, general knowledge, open-ended problems)
• Model pairs from the same family (Qwen, Llama, etc.) for tokenization consistency

When NOT to Use This Skill

• Tasks where all steps equally require high reasoning capability
• Scenarios with extreme latency requirements (RL refinement adds training time)
• Applications requiring guaranteed model invocation (e.g., safety-critical steps)
• Single-model inference (no collaborative pair available)

Problem Summary

Current collaborative approaches use coarse-grained routing, offloading entire queries to large models whenever difficulty emerges. This wastes computational resources since small models typically handle most reasoning steps competently. The small model becomes mere scaffolding—when complexity appears, the entire problem transfers to the expensive model. This approach ignores that reasoning occurs token-by-token: some tokens need teacher guidance while others don't.

Solution: Dynamic Token-Level Teacher Invocation

Train small models to recognize critical tokens and request help only when necessary, with a two-stage training framework.

class RelayLLMStudent:
    def __init__(self, small_model, large_model):
        self.student = small_model
        self.teacher = large_model

    def forward_with_relaying(self, query, max_calls=None):
        """Generate tokens with selective teacher invocation"""
        generated = []
        token_count = 0
        teacher_tokens_used = 0

        while not_finished():
            # Generate next token in student
            token = self.student.generate_next()

            # Check if token is a call command
             token == :
                
                n_tokens = extract_count(token)
                teacher_tokens = .teacher.generate_n_tokens(n_tokens)
                generated.extend(teacher_tokens)
                teacher_tokens_used += n_tokens
                token_count += n_tokens
            :
                
                generated.append(token)
                token_count += 

        efficiency =  - (teacher_tokens_used / token_count)
         generated, efficiency