| name | model-routing |
| description | Use when an agent harness has access to multiple models (Opus/Sonnet/Haiku, or cross-vendor) and must pick the right one per turn based on capability, cost, latency, or load. Apply when designing the routing layer that sits in front of model invocation. Complements graceful-degradation (which covers fallback on failure); this skill covers proactive capability/cost routing. Covers routing signals, classifier design, cost ceilings, and stickiness vs per-turn re-routing. |
Model Routing
Pain Signals — You Need This Pattern When:
- Most turns are simple but a few need top-tier reasoning; using the top model for everything is expensive
- Some turns are latency-critical (user typing) and others are not (background analysis)
- Cost per session varies wildly and most of it goes to easy turns that didn't need premium tiers
- Different agent roles have different needs (planner = capability, executor = cost) — same model is wrong for both
- You have multiple providers / models and want to use whichever is cheapest at adequate quality
Routing is a proactive choice; graceful-degradation is reactive. Routing picks the right model up-front; degradation switches when the chosen model fails. Both layers needed in serious harnesses.
Core Principle
A router is a function (turn_context) → model. It runs before the model call. The choice depends on signals available at that point — and those signals must be cheap to compute.
turn_context ──▶ router ──▶ model_id ──▶ model_call
│
├── signals: complexity estimate, latency budget, $ ceiling,
│ context size, tool-use need, role
│
└── policy: rules / classifier / bandit
The cheaper the routing decision, the more turns can benefit. A 200ms classifier on every turn is fine; a 3s LLM-as-router-judge defeats the point.
Routing Signals
Compute these before routing:
| Signal | How |
|---|
| Task complexity estimate | Heuristic on prompt length, presence of code, math markers, planning verbs |
| Required context size | Sum of system + tools + history + user turn tokens |
| Latency budget | Set by call site (interactive=tight, batch=loose) |
| Cost ceiling | Per-turn $ limit; remaining session budget |
| Tool-use likelihood | Does the turn need tools? (some models cheaper for non-tool turns) |
| Agent role | Planner / executor / reviewer / summarizer — different needs |
| User tier | Free user → cheap; paid → premium |
| Cache locality | Sticking with the same model preserves cache; switching invalidates |
Not all signals are needed for every router. Start with role + complexity; add signals as routing decisions get more refined.
Routing Strategies
Rule-based
routes:
- when: { role: planner }
model: claude-opus-4-7
- when: { role: summarizer, context_tokens: { lt: 8000 } }
model: claude-haiku-4-5
- when: { role: executor, complexity: simple }
model: claude-sonnet-4-6
- default: claude-sonnet-4-6
Cheap, transparent, easy to test. The right starting point.
Classifier-based
Small model (or heuristic) predicts which tier is needed:
def classify_complexity(prompt: str, history: list[Message]) -> Tier:
score = (
0.3 * has_code_blocks(prompt)
+ 0.2 * has_math(prompt)
+ 0.3 * is_open_ended(prompt)
+ 0.2 * (len(history) > 10)
)
return Tier.HIGH if score > 0.6 else Tier.LOW
Or call a small LLM (Haiku) with a fixed cheap prompt to classify. Adds latency; only worth it if rules are insufficient.
Bandit / learned
Multi-armed bandit over (route, outcome). Outcome = (success, cost, latency). Useful when many comparable routes and traffic is high. Operational complexity high — don't reach for it unless rules and classifiers genuinely fall short.
Cache Interaction
Switching models invalidates cache. Each model keeps its own cache line.
Implications:
- Sticky routing within a session for hot path. Cheaper to use one model for many turns than to flip.
- Re-route on natural breakpoints — start of session, after compaction, on explicit role change.
- Avoid per-turn re-routing unless the savings exceed the cache loss.
Estimate routing benefit:
cache_loss = expected_cache_read_tokens × (input_price - cache_read_price)
routing_gain = (current_model_cost - cheaper_model_cost)
route_if = routing_gain > cache_loss
If a session has good cache hit rate, sticky beats opportunistic. If first turn or cold start, route freely.
Escalation
A common pattern: try cheap, escalate on failure.
def try_with_escalation(turn):
cheap_result = call_model(haiku, turn)
if quality_check(cheap_result) == "insufficient":
return call_model(opus, turn)
return cheap_result
Quality check options:
- Self-rated confidence (model emits confidence; threshold check)
- Output validator (schema, length, refusal pattern)
- Downstream usage failure (tool call returned error from bad args; re-route)
Escalation costs both calls when it triggers. Worth it if escalation rate is low (< 20%).
Cost Budget Enforcement
Per-session $ ceiling acts as a hard router constraint:
def route(turn, session):
remaining = session.budget_usd - session.spent_usd
estimated_cost = estimate_cost(turn, primary_model)
if estimated_cost > remaining:
return cheapest_capable_model(turn)
return primary_model
When budget runs out, downgrade or stop — don't silently overspend.
Multi-Provider
If routing across providers (Anthropic, OpenAI, etc.):
- Capability mappings differ; not all features (tool use, vision, caching) work the same
- Keep prompt templates provider-portable, or maintain per-provider variants and route to the right one
- Cost models differ; price-aware routing must use per-provider pricing
- Evaluation must cover all routes; degradation on one provider hides behind average
The complexity rises sharply. Most teams should pick one provider, route within its tiers, and bring multi-provider only when there is a clear gap.
Coordination with Graceful Degradation
| Routing | Graceful Degradation |
|---|
| When | Before call | After failure |
| Trigger | Signals (complexity, budget) | Error, timeout, refusal |
| Goal | Right tool for the job | Keep system running |
| Cache | Choosing same model preserves cache | Fallback model has its own cache |
Pipeline:
turn ──▶ router ──▶ chosen_model ──▶ call ──▶ ok? ──▶ done
│ no
▼
degradation ──▶ fallback model ──▶ ok?
Both layers; don't conflate.
Common Pitfalls
| Pitfall | Symptom | Fix |
|---|
| Per-turn flapping between models | Cache hit rate near zero | Sticky routing within session |
| Routing decision call is itself expensive | Latency added to every turn | Heuristic / small classifier; not LLM-as-judge |
| Capability mismatch unknown until failure | Bad output, retries | Eval per route to know capability boundary |
| No budget enforcement | Cost overruns | Hard ceiling check in router |
| Provider-specific features in shared template | Breaks on routing | Per-provider variants or feature-detect |
| Escalation rate too high | Paying for two calls per turn | Re-tune cheap-tier eligibility |
| Routing logic in many places | Inconsistent decisions | Centralize router; one decision point |
| Hidden routing in middleware | Operators surprised by costs | Surface model in telemetry per turn |
When NOT to Use
- Single-model harnesses — no routing decision exists
- Latency too tight to add even a heuristic — rare; most heuristics are <1ms
- Cost not a concern (small scale) — premium-everything is fine until it isn't
Design Checklist
