scalable-prompt-routing-moe

name	scalable-prompt-routing-moe
title	Scalable Prompt Routing for Frontier LLMs
version	0.0.3
engine	skillxiv-v0.0.3-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.19415
keywords	["Prompt Routing","Mixture of Experts","Graph Clustering","Cost Reduction","Multi-Model Inference"]
description	Route queries across frontier models using two-stage system: graph-based task discovery identifies ~332 latent task types via semantic similarity + preference patterns; MoE with task-specific adapters estimates quality for candidate models. Achieves <50% inference cost of strongest single model while exceeding its performance; applies when managing pools of frontier models with narrow capability gaps.

Component ID

Two-stage LLM prompt routing system combining task discovery and quality estimation.

Motivation

Frontier model pools show narrow performance gaps where subtle capability differences determine task suitability. Routing queries to diverse specialized models can achieve better cost-performance than over-relying on a single strongest model. However, discovering latent task structure and estimating per-model quality requires principled methods beyond heuristics.

The Modification

Stage 1: Graph-Based Task Discovery

Discover latent task types by combining semantic similarity of task descriptions with model preference patterns using Rank Biased Overlap scoring.

# Task discovery via graph clustering on semantic space
def discover_task_types(task_descriptions, model_responses, n_tasks=332):
    """
    Identify latent task types by clustering tasks and models simultaneously.
    Uses semantic embeddings of task descriptions and preference agreement patterns.
    """
    # Embed task descriptions
    task_embeddings = encode_descriptions(task_descriptions)

    # Compute semantic similarity between tasks
    semantic_similarity = cosine_similarity(task_embeddings)

    # Compute preference agreement: which models agree on relative rankings
    model_preferences = []
    for model in available_models:
        rankings = rank_responses(model_responses[model])
        model_preferences.append(rankings)

    # Fuse semantic + preference signals via graph clustering
    combined_graph = semantic_similarity * agreement_matrix(model_preferences)
    task_clusters = spectral_clustering(combined_graph, n_clusters=n_tasks)

    return task_clusters

Stage 2: Task-Specific Quality Estimation

A mixture-of-experts architecture with task-specific prediction heads provides specialized quality estimates alongside general adapters.

# MoE quality estimation with task-specific routing
class TaskAwareQualityMoE:
    """
    Predicts per-model quality for a given task using task-specific adapters.
    Reduces model consideration to ~32% per task (from full 11 models).
    """
     ():
        .task_classifier = TaskClassifier(n_tasks)
        .general_adapters = nn.ModuleList([
            Adapter()  _  (n_models)
        ])
        .task_specific_heads = nn.ModuleList([
            nn.ModuleList([Adapter()  _  (n_models)])
             _  (n_tasks)
        ])

     ():
        
        task_id = .task_classifier(query_embedding)

        
        general_scores = [adapter(query_embedding)  adapter  .general_adapters]

        
        task_specific_scores = [
            .task_specific_heads[task_id][i](query_embedding)
             i  ((.general_adapters))
        ]

        
        final_scores = general_scores + task_specific_scores
         final_scores