ai-model-selector

# AI Model Selector Selects the optimal LLM model for a specific task by analyzing task complexity, latency requirements, cost constraints, and quality expectations. When this skill is active, the model acts as an AI infrastructure architect who evaluates all available models against project requirements and produces a defensible selection with routing strategy. ## TL;DR Checklist - [ ] Decompose task into dimensions: complexity, latency budget, volume, context length - [ ] Evaluate at least 3 candidate models across different providers — never select from one vendor alone - [ ] Separate task capability needs from cost and latency as independent scoring dimensions - [ ] Apply the Task-Type Decision Tree to narrow candidates before scoring - [ ] Build a Capability Scoring Matrix with explicit weights per dimension - [ ] Document benchmark-your-workload principle — leaderboards don't predict real-world performance - [ ] Recommend a fallback model for every selection decision - [ ] Validate provider API availability and rate limits before finalizing --- ## When to Use Use this skill when: - Choosing between GPT-4o, Claude Sonnet/Opus, Gemini 2.5 Pro, or o3 for a coding task and you need a data-driven recommendation - Selecting a model for a RAG pipeline that requires long context windows (>200K tokens) - Deciding on the appropriate cost tier for high-volume classification or summarization workloads - Planning a multi-model fallback architecture with degradation paths - Evaluating open-source models (Llama, Qwen, Mistral) against hosted options for privacy or cost reasons - Comparing latency-sensitive vs. quality-focused model choices for production inference - Justifying model selection to stakeholders with explicit trade-off analysis --- ## When NOT to Use Avoid this skill for: - **Framework selection** — Choose the application framework (LangChain, LlamaIndex, etc.) instead → use `ai-framework-selector` - **Software framework comparison** — Selecting between React, Vue, or Angular → use `framework-selection` - **Already provider-locked scenarios** — When a single vendor contract covers all models and no cross-vendor evaluation is needed - **Model training/fine-tuning decisions** — This skill covers inference-time model selection, not pre-training or fine-tuning strategy - **Hardware/infrastructure provisioning** — Choosing GPUs, TPUs, or cloud instances → that's an infrastructure concern --- ## Core Workflow ``` ┌─────────────────┐ ┌──────────────────────┐ ┌──────────────────┐ │ Task Requirements│────►│ Capability │────►│ Candidate │ │ (complexity, │ │ Classification │ │ Shortlist │ │ latency, cost) │ │ (coding, reasoning, │ │ (3+ cross- │ │ │ │ RAG, creative, etc.) │ │ provider models)│ └─────────────────┘ └──────────────────────┘ └────────┬─────────┘ │ ▼ ┌─────────────────┐ ┌──────────────────────┐ ┌──────────────────┐ │ Fallback Model │◄────│ Selection Decision │◄────│ Capability │ │ Recommendation │ │ (winner, runner-up, │ │ Scoring Matrix │ │ & Degradation │ │ score gap) │ │ (model × dim) │ └─────────────────┘ └──────────────────────┘ └──────────────────┘ ``` 1. **Gather Task Requirements** — Elicit or infer: task type, expected complexity, latency budget, daily request volume, context window needs, output length constraints, and cost ceiling. **Checkpoint:** All five dimensions must have concrete values before proceeding. If any dimension is unknown, flag it explicitly in the output and proceed with assumptions noted. 2. **Classify Task Capabilities** — Map the task to one or more capability categories from the Task-Type Decision Tree (section below). A task may span multiple categories (e.g., coding + long-document review). **Checkpoint:** The primary category drives the initial candidate narrowing; secondary categories inform scoring weights. 3. **Build Candidate Shortlist** — Select at least 3 models across different providers that cover your capability requirements. Never shortlist only from a single vendor ecosystem. Include at least one open-source option if on-premise or cost-sensitive deployment is in scope. **Checkpoint:** If no model meets the minimum latency requirement, escalate to discuss architecture changes (caching, pre-processing, smaller models). 4. **Score Each Candidate** — Build a Capability Scoring Matrix using weighted dimensions: - **Task Fit** (weight 0–1): How well the model excels at this specific capability category - **Latency Adequacy** (weight 0–1): Whether the model's typical p50 latency meets the budget - **Cost Efficiency** (weight 0–1): Cost per million tokens relative to expected volume budget - **Context Window** (weight 0–1): Whether the context window accommodates worst-case input + output - **Ecosystem Maturity** (weight 0–1): SDK support, community size, known production issues **Checkpoint:** The winner must have a score gap of at least 0.15 over the runner-up on the primary dimension. If scores are too close (<0.10 gap), recommend A/B testing with real workload data. 5. **Make Selection Decision** — Choose the model with highest composite score weighted by your priority dimensions. Document the winner, runner-up, and exact score gap. Note any concerns about the selected model's reliability or known limitations. 6. **Design Fallback Architecture** — Select a degradation path: - **Primary → Secondary**: Slightly slower or more expensive alternative for quality fallback - **Primary → Tertiary (fast path)**: Cheaper, faster model for degraded throughput during primary outages - **Circuit breaker**: Define failure thresholds that trigger automatic model switching **Checkpoint:** Every production model selection must include at least one fallback. No exceptions. ### Fallback & Error Routing ``` ┌─────────────────────┐ │ Model API Call │ └──────────┬──────────┘ │ ┌────────────┼────────────┐ ▼ ▼ ▼ Success Rate Limit Server Error (200) (429 / quota) (5xx / timeout) │ │ │ ▼ ▼ ▼ Return Check circuit Retry once with Result breaker state exponential backoff │ │ Open? ──► Yes ──► Fallback Model No ──► │ (retry) ▼ Circuit Opens after N failures │ ▼ Tertiary / Fast-Path Degraded Model ``` **Fallback model rules:** - If primary model hits rate limits: switch to secondary model from same provider (same API key, no re-auth) - If primary model server errors persist (>3 consecutive): activate tertiary fast-path model - If both primary and secondary are unavailable: use cached responses or queue with exponential backoff - Always log fallback activation with timestamp, error type, and duration for post-mortem analysis --- ## 2026 Model Landscape ### Hosted Models **OpenAI:** | Model | Context | Best For | Input / Output ($/1M) | |-------|---------|----------|----------------------| | GPT-4.1 | 1M tokens | Long-document, RAG, coding | $0.60 / $2.40 | | GPT-4.1 Nano | 1M tokens | High-volume classification, cheap long-context | $0.075 / $0.30 | | GPT-4o | 128K tokens | Balanced coding, writing, general-purpose | $2.50 / $10.00 | | GPT-4.1 Mini | 1M tokens | Mid-tier long-context tasks | $0.15 / $0.60 | | o3 | 200K tokens | Complex reasoning, math, proofs | $15.00 / $60.00 | | o3-mini | 200K tokens | Budget reasoning tasks | $1.10 / $4.40 | **Anthropic:** | Model | Context | Best For | Input / Output ($/1M) | |-------|---------|----------|----------------------| | Claude Opus 3.5 | 200K tokens | Complex reasoning, strategic analysis | $15.00 / $75.00 | | Sonnet 4 | 200K tokens | Coding, structured output, tool use | $3.00 / $15.00 | | Sonnet 4 Standard | 200K tokens | Balanced cost/quality (new mid-tier) | $1.25 / $6.25 | | Haiku 4 | 200K tokens | Fast classification, summarization, routing | $0.80 / $4.00 | **Google:** | Model | Context | Best For | Input / Output ($/1M) | |-------|---------|----------|----------------------| | Gemini 2.5 Pro | 1M tokens | Long-document RAG, multimodal, factual QA | $1.25 / $10.00 | | Gemini 2.5 Flash | 1M tokens | Fast long-context tasks, cost-efficient scaling | $0.10 / $0.40 | | Gemini 2.5 Ultra | 1M tokens | Maximum quality multimodal & reasoning | $3.75 / $15.00 | ### Open Source Models (self-hosted via vLLM, TGI, Ollama, Groq) | Model | Params | Best For | Cost (per 1M input, self-hosted) | |-------|--------|----------|---------------------------------| | Llama 3.3 70B | 70B | Best open model per-dollar, coding, reasoning | ~$0.20 (Groq), near-zero (own infra) | | Llama 4 Maverick | ~100B+ | Next-gen open reasoning & multilingual | Varies by deployment | | Qwen 2.5 72B | 72B | Strong multilingual, coding, math | ~$0.30 (Groq) | | Qwen 3 (latest) | Various | Cutting-edge open benchmark leader | Varies | | Mistral Large 2 | 123B | European compliance, multilingual, coding | ~$0.30–$1.00 depending on provider | | Llama 3.2 3B/11B | 3B / 11B | Edge devices, on-device inference | Near-zero (local) | --- ## Task-Type Routing Decision Tree ``` Task Type ──► Primary Recommendation ──► Alternatives ───────────────────────────────────────────────────── Coding & Code Review GPT-4.1 Claude Sonnet 4, o3-mini Claude Sonnet 4 Llama 3.3 70B (self-hosted) Complex Reasoning/Math o3 Claude Opus 3.5 Claude Opus 3.5 Gemini 2.5 Pro Long Document / RAG Gemini 2.5 Pro GPT-4.1 (1M context) (>200K tokens) GPT-4.1 Llama 3.3 + RAG infra Creative/Writing Claude Opus 3.5 GPT-4o, Sonnet 4 GPT-4o Gemini 2.5 Ultra Factual QA w/ Grounding Gemini 2.5 Pro GPT-4.1 GPT-4.1 Claude Sonnet 4 Multimodal (Video) Gemini 2.5 Pro* Gemini 2.5 Ultra only (only Google supports*) Gemini 2.5 Ultra *(as of early 2026, only Google offers native video understanding) Edge / On-Device Llama 3.2 8B Qwen 2.5 7B Qwen 2.5 7B Mistral 7B Instruct Fast Classification/ Haiku 4 GPT-4o-mini, Gemini 2.5 Flash, Routing Llama 3.2 Llama 3.3 on Groq (high throughput) Budget-Critical Llama 3.3 GPT-4.1 Nano, Qwen 2.5, Production 70B self-hosted Gemini 2.5 Flash ``` **Key insight:** The "best" model for any single task-type category varies based on your secondary requirements. A coding task that also requires 500K token context should go to GPT-4.1 over Claude Sonnet 4 despite both being strong coders, simply because the context window requirement is non-negotiable. --- ## Cost-Performance Matrix ``` Budget Tier Models When to Choose ─────────────────────────────────────────────────────────── Ultra-Low GPT-4o-mini ($0.15/M in) >1M daily requests, simple classification (<$0.15/M) Haiku 4 High-volume intent routing Llama 3.2 On-device or air-gapped deployment Low GPT-4.1 Nano ($0.075/M in) Long-context at near-ultra-low cost ($0.08–$0.60/M) Qwen 2.5 Multilingual open-source option Llama 3.3 on Groq High-throughput self-hosted coding Mid GPT-4o ($2.50/M in) General-purpose, balanced quality/cost ($0.60–$3.00/M) Claude Sonnet 4 Structured output, tool use, API reliability Gemini 2.5 Flash Long-context at reasonable cost High Claude Opus 3.5 ($15/M in) Maximum reasoning quality, strategic analysis ($3.00–$15.00/M) o3 Math, proofs, complex multi-step reasoning Gemini 2.5 Pro Multimodal + long-context combined Premium (peak Same as High tier When the task has zero tolerance for errors — performance) budget is secondary to correctness ``` **Cost optimization rule:** Always calculate estimated monthly cost = (input tokens/month × input price) + (output tokens/month × output price). Model selection based on raw per-request quality without volume context is the #1 cause of AI infrastructure budget overruns. --- ## Latency vs. Quality Grid ``` Latency Budget │ Recommended Tier │ Example Models ────────────────┼───────────────────────────────┼──────────────────── <200ms │ Ultra-Low / Low │ GPT-4o-mini, Haiku 4, Llama 3.2 on Groq 200–500ms │ Low / Mid │ GPT-4.1 Nano, Sonnet 4, Gemini 2.5 Flash 500ms–2s │ Mid │ GPT-4o, Claude Sonnet 4, Llama 3.3 70B 2–5s │ High │ Claude Opus 3.5, Gemini 2.5 Pro 5–15s │ Premium (reasoning models) │ o3, o3-mini >15s │ Reasoning only (async) │ o3 complex reasoning mode ``` **Latency note:** These are p50 estimates for text completion at moderate output lengths (~500 tokens). Code generation and tool-calling workflows add 2–5× latency. Always measure p95 latency in your environment — network hops to the provider significantly affect tail latency. --- ## Model Evaluation Example The following demonstrates the scoring methodology. It is a practical reference for building model selection into production systems. ```python from __future__ import annotations import enum import math from dataclasses import dataclass, field from typing import Any, Optional class ModelProvider(str, enum.Enum): """Supported LLM providers.""" OPENAI = "openai" ANTHROPIC = "anthropic" GOOGLE = "google" META_SELFHOSTED = "meta_self_hosted" class TaskCategory(str, enum.Enum): """Primary task categories for model matching.""" CODING = "coding" REASONING = "reasoning" RAG_LONG_CONTEXT = "rag_long_context" CREATIVE_WRITING = "creative_writing" FACTUAL_QA = "factual_qa" MULTIMODAL_VIDEO = "multimodal_video" EDGE_INFERENCE = "edge_inference" FAST_CLASSIFICATION = "fast_classification" @dataclass class TaskRequirements: """Explicit task requirements that drive model selection. All five dimensions must be populated before scoring can proceed. Missing dimensions should use None, which will be handled by the scorer as a neutral (1.0) weight contribution. """ category: TaskCategory max_latency_ms: Optional[int] = None # p50 latency budget max_input_tokens: int = 128_000 # worst-case context window needed daily_volume: Optional[int] = None # estimated requests per day monthly_budget_usd: Optional[float] = None output_length_estimate: int = 500 # expected output tokens per request @dataclass class ModelCandidate: """Represents a candidate model for evaluation.""" name: str provider: ModelProvider input_price_per_1m: float # USD per million input tokens output_price_per_1m: float # USD per million output tokens max_context_tokens: int # context window size typical_latency_ms: float # median p50 latency for standard tasks capability_scores: dict[str, float] # category → quality score (0.0–1.0) @property def is_open_source(self) -> bool: return self.provider == ModelProvider.META_SELFHOSTED @dataclass class CapabilityScore: """Weighted capability breakdown for a model on a task.""" task_fit: float latency_adequacy: float cost_efficiency: float context_window: float ecosystem_maturity: float composite: float def calculate_score( requirements: TaskRequirements, candidate: ModelCandidate, dimension_weights: Optional[dict[str, float]] = None, ) -> CapabilityScore: """Calculate weighted capability score for a model on a specific task. Implements the five-dimension scoring system defined in Core Workflow step 4. Scores are independent — task fit does not influence latency or cost scores, ensuring early-exit evaluation of hard constraints before soft trade-offs. Args: requirements: The task's decomposed requirements across five dimensions. candidate: A ModelCandidate representing a model under consideration. dimension_weights: Optional custom weights (default: equal weighting). Returns: CapabilityScore with per-dimension and composite scores. """ # Default equal weights when no customization provided if dimension_weights is None: dimension_weights = { "task_fit": 0.35, "latency_adequacy": 0.20, "cost_efficiency": 0.15, "context_window": 0.20, "ecosystem_maturity": 0.10, } # --- Hard constraint: context window check (Law 1 — Early Exit) --- if candidate.max_context_tokens < requirements.max_input_tokens: return CapabilityScore( task_fit=0.0, latency_adequacy=0.0, cost_efficiency=0.0, context_window=0.0, ecosystem_maturity=dimension_weights.get("ecosystem_maturity", 0.10), composite=0.0, ) # --- Task fit score (capability match) --- task_fit = candidate.capability_scores.get(requirements.category.value, 0.5) # --- Latency adequacy (how well it fits the latency budget) --- if requirements.max_latency_ms is not None: if candidate.typical_latency_ms <= requirements.max_latency_ms * 0.7: latency_adequacy = 1.0 elif candidate.typical_latency_ms <= requirements.max_latency_ms: latency_adequacy = 0.6 else: # Exceeds budget but not catastrophically — penalize proportionally overshoot_ratio = requirements.max_latency_ms / candidate.typical_latency_ms