agent-as-a-router-coding

name	agent-as-a-router-coding
description	Use Agent-as-a-Router (ACRouter) to intelligently route coding tasks to optimal models under performance-cost tradeoffs
triggers	["route this coding task to the best model","use ACRouter to select a model for this problem","set up agent-as-a-router for my project","evaluate models with CodeRouterBench","integrate ACRouter into my coding workflow","run the ACRouter baselines and benchmarks","implement agentic model routing for code tasks","add intelligent model selection to my agent"]

Agent-as-a-Router Coding Skill

Skill by ara.so — AI Agent Skills collection.

ACRouter is an agentic model routing system that intelligently selects backend models for coding tasks, balancing performance and cost. It uses verifier feedback and escalation strategies to route problems through a hierarchy of models (cheap → strong), stopping when a solution passes verification. This skill covers installation, reproduction of benchmark results, runtime integration, and custom inference patterns.

What ACRouter Does

• Agentic Routing: Routes coding tasks to different models (cheap-first, escalate on failure)
• CodeRouterBench: Public benchmark with ID (in-distribution) and OOD176 (out-of-distribution) tasks
• Verifier-Driven: Uses test execution or static analysis to validate solutions before escalating
• Cost-Performance Tradeoff: Optimizes for high performance per dollar spent
• Runtime Integration: Ships with plugins for Claude Code Router, cc-switch, and generic OpenRouter-compatible APIs

Installation

# Clone the repository
git clone https://github.com/LanceZPF/agent-as-a-router.git
cd agent-as-a-router

# Create conda environment
conda create -n acrouter python=3.11 -y
conda activate acrouter

# Install dependencies
python -m pip install --upgrade pip setuptools wheel
python -m pip install -r requirements.txt
python -m pip install -e .

# Run tests to verify installation
python -m unittest discover -s tests

Reproduce Benchmark Results

ID (In-Distribution) Evaluation

python scripts/run_id.py --output-dir outputs/tmp/id

Expected output: ID n=2919 AvgPerf=50.14 CumReg=202.0 $Total=22.31 Perf/$=2.25 rAcc=0.2395

OOD176 (Out-of-Distribution) with ACRouter

python scripts/run_acrouter_ood176.py --output-dir outputs/tmp/acrouter_ood176

Expected output: ACRouter-OOD176 n=176 AvgPerf=73.30 CumReg=15.9 $Total=86.72 Perf/$=0.85

OOD176 Baselines Comparison

python scripts/run_baselines_ood176.py --output-dir outputs/tmp/baselines_ood176

Generates a comparison table with Oracle, Single-Model, Round-Robin, and ACRouter strategies.

Download Hugging Face Assets

Minimal Dataset (OOD176 replay only)

python scripts/download_hf_assets.py --minimal --dataset-dir .hf/CodeRouterBench

With Optional Trained Router Model

python scripts/download_hf_assets.py \
  --minimal \
  --with-router-model \
  --dataset-dir .hf/CodeRouterBench \
  --model-dir .hf/router_model

Run from Downloaded Snapshot

python scripts/run_acrouter_ood176.py \
  --hf-dataset-dir .hf/CodeRouterBench \
  --output-dir outputs/tmp/acrouter_ood176_hf

python scripts/run_baselines_ood176.py \
  --hf-dataset-dir .hf/CodeRouterBench \
  --output-dir outputs/tmp/baselines_ood176_hf

Runtime Integration: Inference API

Basic ACRouter Usage

from acrouter_repro.inference import ACRouter

# Initialize router with model hierarchy
router = ACRouter(
    candidate_models=["gpt-4o-mini", "gpt-4o", "claude-3.5-sonnet"],
    cheap_chain=["gpt-4o-mini"],
    escalate_to="gpt-4o",
    k=1,  # Number of cheap attempts before escalation
)

# Define your backend model caller
def call_model(model: str, task: dict) -> str:
    """Call your actual model API (OpenRouter, OpenAI, etc.)"""
    # Example: use OpenRouter
    import openai
    client = openai.OpenAI(
        base_url="https://openrouter.ai/api/v1",
        api_key=os.environ["OPENROUTER_API_KEY"]
    )
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": task["prompt"]}]
    )
    return response.choices[0].message.content

# Define your verifier (tests, static analysis, etc.)
def verify_solution(response: str, task: dict, model: str) -> bool:
    """Validate the generated code"""
    # Example: run pytest or static checks
    # Return True if solution passes, False to escalate
    return run_tests(response, task["test_file"])

# Route a task
task = {
    "task_id": "two_sum",
    "dimension": "algorithm",
    "prompt": "Write a function that solves two-sum problem...",
    "test_file": "tests/test_two_sum.py"
}

decision = router.run_with_verifier(
    task=task,
    call_model=call_model,
    verify=verify_solution
)

print(f"Chosen model: {decision.chosen_model}")
print(f"Solution: {decision.final_response}")
print(f"Cost: ${decision.total_cost:.4f}")

Complete Inference Example

import os
from acrouter_repro.inference import ACRouter

def main():
    # Set up router
    router = ACRouter(
        candidate_models=["deepseek-coder-v2", "claude-3.5-sonnet"],
        cheap_chain=["deepseek-coder-v2"],
        escalate_to="claude-3.5-sonnet",
        k=2  # Try cheap model twice before escalating
    )
    
    # Mock model caller (replace with real API)
    def call_model(model: str, task: dict) -> str:
        print(f"[ACRouter] Calling {model}...")
        # Your actual API call here
        return f"def solution(): pass  # Generated by {model}"
    
    # Mock verifier (replace with real test runner)
    def verify(response: str, task: dict, model: str) -> bool:
        print(f"[ACRouter] Verifying solution from {model}...")
        # Run actual tests: subprocess.run(["pytest", task["test_file"]])
        return "claude" in model  # Mock: only strong model passes
    
    # Task definition
    task = {
        "task_id": "bug_fix_001",
        "dimension": "bug_fixing",
        "prompt": "Fix the null pointer exception in src/parser.py"
    }
    
    # Route and solve
    decision = router.run_with_verifier(
        task=task,
        call_model=call_model,
        verify=verify
    )
    
    print(f"\n[Result]")
    print(f"  Model: {decision.chosen_model}")
    print(f"  Success: {decision.verified}")
    print(f"  Attempts: {len(decision.attempt_history)}")

if __name__ == "__main__":
    main()

Run: python examples/inference_demo.py

Demo: API Coding Solver

Route a programming problem through multiple models until verification passes.

Setup

export OPENROUTER_API_KEY="your-key-here"

Configuration

Create demos/api_coding_solver/models.json:

{
  "models": [
    {
      "name": "deepseek/deepseek-coder",
      "cost_per_1k_tokens": 0.0002,
      "provider": "openrouter"
    },
    {
      "name": "anthropic/claude-3.5-sonnet",
      "cost_per_1k_tokens": 0.015,
      "provider": "openrouter"
    }
  ],
  "verifier": {
    "command": "python",
    "args": ["-m", "pytest", "--tb=short"]
  }
}

Run with Dry-Run

python demos/api_coding_solver/solve.py \
  --config demos/api_coding_solver/models.example.json \
  --problem-file demos/api_coding_solver/problems/two_sum.txt \
  --dry-run

Solve a Problem

python demos/api_coding_solver/solve.py \
  --config demos/api_coding_solver/models.example.json \
  --problem-file demos/api_coding_solver/problems/two_sum.txt

Demo: Commercial CLI Router

Route prompts to Codex, Claude Code, or Opencode CLI tools.

Setup

# Set command prefixes (optional wrappers)
export ACROUTER_CODEX_PREFIX="ccswitch codex --"
export ACROUTER_CLAUDE_PREFIX="ccswitch claude --"
export ACROUTER_OPENCODE_PREFIX="ccswitch opencode --"

Configuration

Edit demos/commercial_cli_router/tools.example.json:

{
  "tools": {
    "codex": {
      "command": "codex",
      "args": ["--workdir", "{workdir}", "--prompt", "{prompt}"]
    },
    "claude": {
      "command": "claude-code",
      "args": ["--cwd", "{workdir}", "{prompt}"]
    },
    "opencode": {
      "command": "opencode",
      "args": ["{prompt}", "--directory", "{workdir}"]
    }
  },
  "default_tool": "codex"
}

Route a Prompt

# Dry-run (show command without execution)
python demos/commercial_cli_router/router_mvp.py \
  --prompt "Patch this repository so pytest passes" \
  --dry-run

# Execute with selected tool
python demos/commercial_cli_router/router_mvp.py \
  --tool codex \
  --workdir /path/to/project \
  --prompt "Run the tests and fix the failing parser case"

Config-Driven Pipeline

Use when you have precomputed task/model results.

Example Config

Create configs/my_eval.json:

{
  "input": {
    "matrix_file": "data/matrices/phase2_ood/unified/matrix_acrouter_ood176.json"
  },
  "router": {
    "type": "acrouter",
    "candidate_models": ["gpt-4o-mini", "gpt-4o"],
    "cheap_chain": ["gpt-4o-mini"],
    "escalate_to": "gpt-4o",
    "k": 1
  },
  "output": {
    "dir": "outputs/my_eval",
    "formats": ["csv", "json", "table"]
  }
}

Run Pipeline

python scripts/run_pipeline.py --config configs/my_eval.json

Add Custom Benchmark

Prepare Input Data

Create my_tasks.jsonl:

{"task_id": "task_001", "dimension": "bug_fixing", "prompt": "Fix the parser..."}
{"task_id": "task_002", "dimension": "feature", "prompt": "Add CSV export..."}

Create my_results.jsonl:

{"task_id": "task_001", "model": "gpt-4o-mini", "resolved": true, "input_tokens": 150, "output_tokens": 300}
{"task_id": "task_001", "model": "gpt-4o", "resolved": true, "input_tokens": 150, "output_tokens": 280}
{"task_id": "task_002", "model": "gpt-4o-mini", "resolved": false, "input_tokens": 200, "output_tokens": 150}
{"task_id": "task_002", "model": "gpt-4o", "resolved": true, "input_tokens": 200, "output_tokens": 400}

Pipeline Config

{
  "input": {
    "tasks_file": "my_tasks.jsonl",
    "results_file": "my_results.jsonl"
  },
  "router": {
    "type": "acrouter",
    "candidate_models": ["gpt-4o-mini", "gpt-4o"],
    "cheap_chain": ["gpt-4o-mini"],
    "escalate_to": "gpt-4o",
    "k": 1
  },
  "output": {