agent-as-a-router-coding

# Agent-as-a-Router Coding Skill > Skill by [ara.so](https://ara.so) — AI Agent Skills collection. ACRouter is an agentic model routing system that intelligently selects backend models for coding tasks, balancing performance and cost. It uses verifier feedback and escalation strategies to route problems through a hierarchy of models (cheap → strong), stopping when a solution passes verification. This skill covers installation, reproduction of benchmark results, runtime integration, and custom inference patterns. ## What ACRouter Does - **Agentic Routing**: Routes coding tasks to different models (cheap-first, escalate on failure) - **CodeRouterBench**: Public benchmark with ID (in-distribution) and OOD176 (out-of-distribution) tasks - **Verifier-Driven**: Uses test execution or static analysis to validate solutions before escalating - **Cost-Performance Tradeoff**: Optimizes for high performance per dollar spent - **Runtime Integration**: Ships with plugins for Claude Code Router, cc-switch, and generic OpenRouter-compatible APIs ## Installation ```bash # Clone the repository git clone https://github.com/LanceZPF/agent-as-a-router.git cd agent-as-a-router # Create conda environment conda create -n acrouter python=3.11 -y conda activate acrouter # Install dependencies python -m pip install --upgrade pip setuptools wheel python -m pip install -r requirements.txt python -m pip install -e . # Run tests to verify installation python -m unittest discover -s tests ``` ## Reproduce Benchmark Results ### ID (In-Distribution) Evaluation ```bash python scripts/run_id.py --output-dir outputs/tmp/id ``` Expected output: `ID n=2919 AvgPerf=50.14 CumReg=202.0 $Total=22.31 Perf/$=2.25 rAcc=0.2395` ### OOD176 (Out-of-Distribution) with ACRouter ```bash python scripts/run_acrouter_ood176.py --output-dir outputs/tmp/acrouter_ood176 ``` Expected output: `ACRouter-OOD176 n=176 AvgPerf=73.30 CumReg=15.9 $Total=86.72 Perf/$=0.85` ### OOD176 Baselines Comparison ```bash python scripts/run_baselines_ood176.py --output-dir outputs/tmp/baselines_ood176 ``` Generates a comparison table with Oracle, Single-Model, Round-Robin, and ACRouter strategies. ## Download Hugging Face Assets ### Minimal Dataset (OOD176 replay only) ```bash python scripts/download_hf_assets.py --minimal --dataset-dir .hf/CodeRouterBench ``` ### With Optional Trained Router Model ```bash python scripts/download_hf_assets.py \ --minimal \ --with-router-model \ --dataset-dir .hf/CodeRouterBench \ --model-dir .hf/router_model ``` ### Run from Downloaded Snapshot ```bash python scripts/run_acrouter_ood176.py \ --hf-dataset-dir .hf/CodeRouterBench \ --output-dir outputs/tmp/acrouter_ood176_hf python scripts/run_baselines_ood176.py \ --hf-dataset-dir .hf/CodeRouterBench \ --output-dir outputs/tmp/baselines_ood176_hf ``` ## Runtime Integration: Inference API ### Basic ACRouter Usage ```python from acrouter_repro.inference import ACRouter # Initialize router with model hierarchy router = ACRouter( candidate_models=["gpt-4o-mini", "gpt-4o", "claude-3.5-sonnet"], cheap_chain=["gpt-4o-mini"], escalate_to="gpt-4o", k=1, # Number of cheap attempts before escalation ) # Define your backend model caller def call_model(model: str, task: dict) -> str: """Call your actual model API (OpenRouter, OpenAI, etc.)""" # Example: use OpenRouter import openai client = openai.OpenAI( base_url="https://openrouter.ai/api/v1", api_key=os.environ["OPENROUTER_API_KEY"] ) response = client.chat.completions.create( model=model, messages=[{"role": "user", "content": task["prompt"]}] ) return response.choices[0].message.content # Define your verifier (tests, static analysis, etc.) def verify_solution(response: str, task: dict, model: str) -> bool: """Validate the generated code""" # Example: run pytest or static checks # Return True if solution passes, False to escalate return run_tests(response, task["test_file"]) # Route a task task = { "task_id": "two_sum", "dimension": "algorithm", "prompt": "Write a function that solves two-sum problem...", "test_file": "tests/test_two_sum.py" } decision = router.run_with_verifier( task=task, call_model=call_model, verify=verify_solution ) print(f"Chosen model: {decision.chosen_model}") print(f"Solution: {decision.final_response}") print(f"Cost: ${decision.total_cost:.4f}") ``` ### Complete Inference Example ```python import os from acrouter_repro.inference import ACRouter def main(): # Set up router router = ACRouter( candidate_models=["deepseek-coder-v2", "claude-3.5-sonnet"], cheap_chain=["deepseek-coder-v2"], escalate_to="claude-3.5-sonnet", k=2 # Try cheap model twice before escalating ) # Mock model caller (replace with real API) def call_model(model: str, task: dict) -> str: print(f"[ACRouter] Calling {model}...") # Your actual API call here return f"def solution(): pass # Generated by {model}" # Mock verifier (replace with real test runner) def verify(response: str, task: dict, model: str) -> bool: print(f"[ACRouter] Verifying solution from {model}...") # Run actual tests: subprocess.run(["pytest", task["test_file"]]) return "claude" in model # Mock: only strong model passes # Task definition task = { "task_id": "bug_fix_001", "dimension": "bug_fixing", "prompt": "Fix the null pointer exception in src/parser.py" } # Route and solve decision = router.run_with_verifier( task=task, call_model=call_model, verify=verify ) print(f"\n[Result]") print(f" Model: {decision.chosen_model}") print(f" Success: {decision.verified}") print(f" Attempts: {len(decision.attempt_history)}") if __name__ == "__main__": main() ``` Run: `python examples/inference_demo.py` ## Demo: API Coding Solver Route a programming problem through multiple models until verification passes. ### Setup ```bash export OPENROUTER_API_KEY="your-key-here" ``` ### Configuration Create `demos/api_coding_solver/models.json`: ```json { "models": [ { "name": "deepseek/deepseek-coder", "cost_per_1k_tokens": 0.0002, "provider": "openrouter" }, { "name": "anthropic/claude-3.5-sonnet", "cost_per_1k_tokens": 0.015, "provider": "openrouter" } ], "verifier": { "command": "python", "args": ["-m", "pytest", "--tb=short"] } } ``` ### Run with Dry-Run ```bash python demos/api_coding_solver/solve.py \ --config demos/api_coding_solver/models.example.json \ --problem-file demos/api_coding_solver/problems/two_sum.txt \ --dry-run ``` ### Solve a Problem ```bash python demos/api_coding_solver/solve.py \ --config demos/api_coding_solver/models.example.json \ --problem-file demos/api_coding_solver/problems/two_sum.txt ``` ## Demo: Commercial CLI Router Route prompts to Codex, Claude Code, or Opencode CLI tools. ### Setup ```bash # Set command prefixes (optional wrappers) export ACROUTER_CODEX_PREFIX="ccswitch codex --" export ACROUTER_CLAUDE_PREFIX="ccswitch claude --" export ACROUTER_OPENCODE_PREFIX="ccswitch opencode --" ``` ### Configuration Edit `demos/commercial_cli_router/tools.example.json`: ```json { "tools": { "codex": { "command": "codex", "args": ["--workdir", "{workdir}", "--prompt", "{prompt}"] }, "claude": { "command": "claude-code", "args": ["--cwd", "{workdir}", "{prompt}"] }, "opencode": { "command": "opencode", "args": ["{prompt}", "--directory", "{workdir}"] } }, "default_tool": "codex" } ``` ### Route a Prompt ```bash # Dry-run (show command without execution) python demos/commercial_cli_router/router_mvp.py \ --prompt "Patch this repository so pytest passes" \ --dry-run # Execute with selected tool python demos/commercial_cli_router/router_mvp.py \ --tool codex \ --workdir /path/to/project \ --prompt "Run the tests and fix the failing parser case" ``` ## Config-Driven Pipeline Use when you have precomputed task/model results. ### Example Config Create `configs/my_eval.json`: ```json { "input": { "matrix_file": "data/matrices/phase2_ood/unified/matrix_acrouter_ood176.json" }, "router": { "type": "acrouter", "candidate_models": ["gpt-4o-mini", "gpt-4o"], "cheap_chain": ["gpt-4o-mini"], "escalate_to": "gpt-4o", "k": 1 }, "output": { "dir": "outputs/my_eval", "formats": ["csv", "json", "table"] } } ``` ### Run Pipeline ```bash python scripts/run_pipeline.py --config configs/my_eval.json ``` ## Add Custom Benchmark ### Prepare Input Data Create `my_tasks.jsonl`: ```jsonl {"task_id": "task_001", "dimension": "bug_fixing", "prompt": "Fix the parser..."} {"task_id": "task_002", "dimension": "feature", "prompt": "Add CSV export..."} ``` Create `my_results.jsonl`: ```jsonl {"task_id": "task_001", "model": "gpt-4o-mini", "resolved": true, "input_tokens": 150, "output_tokens": 300} {"task_id": "task_001", "model": "gpt-4o", "resolved": true, "input_tokens": 150, "output_tokens": 280} {"task_id": "task_002", "model": "gpt-4o-mini", "resolved": false, "input_tokens": 200, "output_tokens": 150} {"task_id": "task_002", "model": "gpt-4o", "resolved": true, "input_tokens": 200, "output_tokens": 400} ``` ### Pipeline Config ```json { "input": { "tasks_file": "my_tasks.jsonl", "results_file": "my_results.jsonl" }, "router": { "type": "acrouter", "candidate_models": ["gpt-4o-mini", "gpt-4o"], "cheap_chain": ["gpt-4o-mini"], "escalate_to": "gpt-4o", "k": 1 }, "output": {