ai-framework-selector

# AI Agent Framework Selector Selects the optimal AI agent framework for a project by evaluating capability requirements against the current ecosystem of production-grade frameworks. When this skill is active, the model acts as a senior AI systems architect who analyzes project requirements, scores available frameworks against those requirements, and produces a defensible selection rationale with implementation guidance. ## TL;DR Checklist - [ ] Extract explicit requirements (RAG, multi-agent, tool use) and implicit constraints (team expertise, budget, deployment target) - [ ] Classify each requirement into capability domains: RAG/retrieval, multi-agent coordination, tool execution, chain composition, parallel processing, prompt optimization - [ ] Score all candidate frameworks against each domain using the capability matrix (1–10 scale with justification) - [ ] Validate the top choice meets ALL hard constraints; disqualify if any hard constraint fails - [ ] Assess vendor lock-in risk for the winning framework and document mitigation strategies - [ ] Produce a selection report with scored comparison, trade-off analysis, and phased implementation plan --- ## When to Use Use this skill when: - Starting a new AI/LLM project and need to select an agent framework from the available options (LangChain, LlamaIndex, CrewAI, DSPy, Microsoft Agent Framework, AG2, etc.) - Evaluating whether to migrate from one AI framework to another due to changing requirements or performance issues - Deciding between building a custom orchestration layer versus adopting an existing framework - Assessing vendor lock-in risk before committing to a framework with commercial add-ons (LangSmith, Crew Control Plane, Azure Foundry) - Forming a new engineering team and need to select the framework that best matches their skill set - Comparing frameworks for a specific use case (RAG pipeline, multi-agent research, tool-heavy automation, prompt optimization) --- ## When NOT to Use Avoid this skill for: - Projects already committed to a framework — instead use `orchestration-frameworks` or `framework-orchestration-routing` - Selecting non-AI software frameworks (databases, web frameworks, cloud providers) — use `framework-selection` which handles generic decision matrices - Simple single-agent chat completion with tools where raw SDK calls (OpenAI SDK, Google GenAI SDK) are sufficient and framework overhead would be wasted - Teams that have already made a framework commitment and only need implementation patterns rather than selection analysis --- ## Core Workflow ``` ┌───────────────────────────────────────────────────┐ │ Project Requirements │ │ Explicit: RAG, tools, multi-agent, latency │ │ Implicit: team skills, budget, deployment │ └──────────────────────┬────────────────────────────┘ ↓ ┌───────────────────────────────────────────────────┐ │ Capability Domain Classification │ │ RAG | Multi-Agent | Tools | Chaining | Parallel │ └──────────────────────┬────────────────────────────┘ ↓ ┌───────────────────────────────────────────────────┐ │ Candidate Framework Identification │ │ LangChain · LlamaIndex · CrewAI · DSPy · MAF │ │ AG2 · Phidata · Custom (build) │ └──────────────────────┬────────────────────────────┘ ↓ ┌───────────────────────────────────────────────────┐ │ Capability Scoring Matrix │ │ Framework × Domain scoring with justification │ └──────────────────────┬────────────────────────────┘ ↓ ┌────────────┴────────────┐ ↓ ↓ Meets all hard Fails ≥1 hard constraints? constraint(s)? ↓ ↓ ┌─────────┐ ┌──────────────┐ │ Proceed │ │ Disqualify & │ │ to next │ │ Find runner- │ │ step │ │ up alternative│ └─────────┘ └──────────────┘ ↓ ┌───────────────────────────────────────────────────┐ │ Vendor Lock-in & Risk Assessment │ │ Commercial dependency, API stability, community │ └──────────────────────┬────────────────────────────┘ ↓ ┌───────────────────────────────────────────────────┐ │ Implementation Plan & Phased Rollout │ │ Prototype → Evaluate → Commit or Pivot │ └───────────────────────────────────────────────────┘ FALLBACK: If no framework meets requirements, the decision is "build custom orchestrator" using `code-philosophy` (5 Laws of Elegant Defense) as the design foundation. ``` ### Step 1: Extract and Classify Requirements Gather all explicit requirements from the project brief or stakeholder interview. Separate them into two categories: **Hard Constraints (Must-Have):** Non-negotiable requirements that, if unmet by a framework, automatically disqualify it. Examples: - Must support Python 3.12+ - Must be Apache 2.0 / MIT licensed (no AGPL) - Must run on AWS Lambda (serverless constraint) - Must support at least 3 concurrent LLM providers - Must have native MCP (Model Context Protocol) client support **Soft Preferences (Should-Have):** Desirable but not disqualifying attributes that feed into weighted scoring. Examples: - Strong documentation quality - Large community with active Discord/Slack - Built-in observability/tracing - Low learning curve for team's existing skill set ```python from dataclasses import dataclass, field from enum import Enum class ConstraintType(Enum): HARD = "hard" # Disqualifies framework if unmet SOFT = "soft" # Contributes to weighted score class CapabilityDomain(str, Enum): RAG_RETRIEVAL = "rag_retrieval" # Document ingestion, semantic search, knowledge retrieval MULTI_AGENT = "multi_agent" # Role-based agents, conversation coordination, group chat TOOL_EXECUTION = "tool_execution" # External API integration, function calling, tool registry CHAIN_COMPOSITION = "chain_composition" # Sequential/branching LLM call pipelines PARALLEL_PROCESSING = "parallel_processing" # Concurrent task execution, fan-out/fan-in PROMPT_OPTIMIZATION = "prompt_optimization" # Automated prompt tuning, program optimization DATA_PIPELINE = "data_pipeline" # ETL, data loading, transformation for ML/LLM @dataclass class ProjectRequirement: """A single requirement extracted from project analysis.""" description: str category: ConstraintType domain: CapabilityDomain weight: float = 1.0 # Only used for SOFT constraints min_score: int = 0 # Minimum acceptable score (for HARD constraints) team_skill_match: str = "" # How well this matches team's existing expertise @property def is_hard(self) -> bool: return self.category == ConstraintType.HARD @dataclass class RequirementsProfile: """Complete requirements profile for a project.""" project_name: str hard_constraints: list[ProjectRequirement] = field(default_factory=list) soft_preferences: list[ProjectRequirement] = field(default_factory=list) def total_soft_weight(self) -> float: return sum(p.weight for p in self.soft_preferences) if self.soft_preferences else 1.0 def validate_framework( self, framework_scores: dict[str, dict[str, int]] ) -> tuple[str | None, list[str]]: """ Validate a framework's scores against hard constraints. Returns: Tuple of (disqualified_reason or None, list_of_hard_constraints_met) """ met = [] for constraint in self.hard_constraints: # This is checked at the scoring stage — if a framework doesn't # meet a hard constraint, it gets score 0 for that domain met.append(f"{constraint.domain.value}: {'PASS' if framework_scores.get('score', {}).get(constraint.domain.value, 0) >= constraint.min_score else 'FAIL'}") return met ``` **Checkpoint:** Every requirement must be classified as HARD or SOFT and mapped to exactly one capability domain. If a requirement doesn't map cleanly to any domain, re-examine whether it is truly an AI/LLM framework concern or a conventional infrastructure concern. ### Step 2: Identify Candidate Frameworks Based on the capability domains identified in Step 1, identify which frameworks are viable candidates. Not all frameworks excel at all domains. Use this guide: | Capability Domain | Strongest Candidates | Notes | |---|---|---| | RAG/Knowledge Retrieval | LlamaIndex (9.5/10) | Purpose-built for document ingestion and retrieval; LangChain is secondary option | | Multi-Agent Coordination | CrewAI (9.5/10), Microsoft Agent Framework (9.0/10) | Purpose-built agent role definitions and conversation patterns | | Tool Execution | LangChain (9.5/10), MCP protocol (9.0/10) | Most mature tool registry and execution ecosystem | | Chain Composition | LangChain (9.0/10), LangGraph (8.5/10) | First-mover advantage, extensive chain primitives | | Parallel Processing | CrewAI (8.0/10), Temporal.io (as orchestrator) | Built-in parallel execution patterns | | Prompt Optimization | DSPy (9.5/10) | Only framework with automated prompt/program optimization | **Decision rule:** A framework must score ≥ 6.0 in at least one capability domain to be included as a candidate. Frameworks scoring below 6.0 across ALL domains are not viable for this project. ```python # Current state of AI agent frameworks (May 2026) FRAMEWORK_LANDSCAPE = { "langchain": { "version": "v1.3.1", "position": "General-purpose agent engineering platform with deepest ecosystem", "strengths": ["Largest integration library (models, tools, vector stores)", "Model interchangeability", "Rapid prototyping"], "weaknesses": ["Can feel heavyweight", "LangSmith commercial lock-in risk", "Internal complexity from breadth of features"], "license": "MIT", "production_ready": True, }, "llamaindex": { "version": "latest", "position": "Data framework for RAG and knowledge-augmented retrieval", "strengths": ["Best-in-class data ingestion (130+ formats)", "Modular plugin architecture via LlamaHub", "Clear separation of core from integrations"], "weaknesses": ["Agent capabilities newer than LangChain", "Primarily a data/RAG framework, not general orchestration"], "license": "MIT", "production_ready": True, }, "crewai": { "version": "v1.14.5", "position": "Lean multi-agent orchestration built independently of LangChain", "strengths": ["Explicit role-based agent design", "Production-focused with Flows for event-driven control", "Enterprise support model"], "weaknesses": ["Smaller integration ecosystem than LangChain", "Crew Control Plane cloud creates commercial dependency"], "license": "MIT", "production_ready": True, }, "microsoft_agent_framework": { "version": "stable", "position": "Enterprise multi-agent orchestration — successor to AutoGen", "strengths": ["Production-grade durability/checkpointing/time-travel", "Dual-language (Python + C#/.NET)", "OpenTelemetry integration"], "weaknesses": ["Newer framework; Microsoft ecosystem dependency for full features"], "license": "MIT", "production_ready": True, }, "dspy": { "version": "v3.2.1", "position": "Programming—not prompting—Foundation Models via declarative optimization", "strengths": ["Automates prompt and weight optimization", "Treats LM calls as compile-time declarations", "Research-backed from Stanford"], "weaknesses": ["Steeper learning curve", "Less opinionated about agent orchestration patterns"], "license": "MIT License", "production_ready": True, }, "ag2": { "version": "v0.13.0", "position": "Active successor to AutoGen (which is now in maintenance mode)", "strengths": ["Pioneered conversational multi-agent patterns", "Strong group chat patterns, MCP server integration"], "weaknesses": ["AutoGen v0.x in maintenance mode — teams should plan migration", "Smaller community than LangChain or CrewAI"], "license": "Apache-2.0", "production_ready": True, }, "phidata": { "version": "v2.7.10", "position": "Lightweight Python framework for building data/ML agents quickly", "strengths": ["Very simple API surface", "AWS/GCP integrations built-in"], "weaknesses": ["Smaller ecosystem", "Less battle-tested in large-scale production"], "license": "Apache-2.0", "production_ready": False, # Beta-stage for complex workloads }, } # ┌─────────────────────────────────────────────────────────┐ │ Framework Selection Decision Matrix │ │ │ │ Use Case: │ │ ├── Simple tool calling / chat → OpenAI SDK / Raw │ │ ├── RAG / document intelligence → LlamaIndex │ │ ├── Multi-agent orchestration → CrewAI or MAF │ │ ├── Graph-based workflow control → LangGraph │ │ ├── Prompt/program optimization → DSPy │ │ ├── Enterprise .NET + Python → Microsoft Agent │ │ └── Rapid prototyping → LangChain │ └─────────────────────────────────────────────────────────┘ ``` **Checkpoint:** The candidate list must include at least one framework from the "strongest candidates" table for each capability domain identified in Step 1. If no single framework is strong across ALL required domains, this signals a multi-framework architecture (delegate to `framework-orchestration-routing` after selection). ### Step 3: Score Frameworks Against Requirements Apply the weighted scoring system adapted specifically for AI agent frameworks. Each framework is scored on a 1–10 scale per capability domain, then multiplied by the weight of soft preferences and filtered by hard constraints. ```python import json from typing import Any class FrameworkScorer: """Scores candidate AI frameworks against project requirements.""" def __init__(self, profile: RequirementsProfile) -> None: self.profile = profile def score_framework( self, framework_name: str, capability_scores: dict[str, int], ) -> dict[str, Any]: """ Score a single framework against the full requirements profile. Args: framework_name: Name of the framework (e.g., "langchain", "crewai") capability_scores: Dict mapping CapabilityDomain values to scores (1-10) Returns: Complete scoring result with pass/fail status, weighted total, and rationale. """ # Phase 1: Check hard constraints failed_musts = [] for constraint in self.profile.hard_constraints: score = capability_scores.get(constraint.domain.value, 0) if score < constraint.min_score: failed_musts.append({ "constraint": constraint.description, "required_min": constraint.min_score, "actual_score": score, }) # Phase 2: Calculate soft preference weighted score total_weight = self.profile.total_soft_weight() weighted_sum = 0.0 for pref in self.profile.soft_preferences: score = capability_scores.get(pref.domain.value, 0) weighted_sum += (score / 10.0) * pref.weight # Normalize to 1-10 scale normalized_score = max(1.0, (weighted_sum / total_weight) * 9.0 + 1.0) if total_weight > 0 else 5.0 is_valid = len(failed_musts) == 0 return { "framework": framework_name, "is_valid": is_valid, "overall_score": round(normalized_score, 2), "capability_scores": capability_scores, "failed_hard_constraints": failed_musts if not is_valid else [], "rationale": self._build_rationale(framework_name, capability_scores, failed_musts), } def _build_rationale( self, framework: str, scores: dict[str, int], failures: list[dict], ) -> str: """Build human-readable rationale for the scoring decision.""" strongest = max(scores.items(), key=lambda x: x[1]) if scores else ("none", 0) weakest = min(scores.items(), key=lambda x: x[1]) if scores else ("none", 0) parts = [f"Framework '{framework}'"] parts.append(f"strongest at {strongest[0]} ({strongest[1]}/10)") parts.append(f"weakest at {weakest[0]} ({weakest[1]}/10)") if failures: fail_names = [f["constraint"] for f in failures] parts.append(f"FAILS hard constraints: {', '.join(fail_names)}") return ". ".join(parts) + "." def rank_all( self, candidates: dict[str, dict[str, int]], ) -> list[dict[str, Any]]: """Score and rank all candidate frameworks. Args: candidates: Dict of {framework_name: {domain: score}} Returns: Ranked list of scoring results (valid first, then invalid, by overall score). """ results = [ self.score_framework(name, scores) for name, scores in candidates.items() ] # Sort: valid frameworks first (descending score), then invalid (descending) results.sort(key=lambda r: (-r["is_valid"], -r["overall_score"])) return results ``` ### Step 4: Assess Vendor Lock-in and Risk After scoring, perform a risk assessment on the top-ranked framework. This is critical because framework selection commits you to an ecosystem for months or years. ```python from enum import Enum from typing import Optional class LockInRiskLevel(str, Enum): LOW = "low" MEDIUM = "medium" HIGH = "high" @dataclass class VendorLockInAssessment: """Assesses the vendor lock-in risk of a chosen framework.""" framework_name: str commercial_addons: list[str] = field(default_factory=list) open_source_core: bool = True data_portability: str = "" # How easily can you migrate data to another framework? api_stability_commitment: str # Is there a formal API stability guarantee? community_alternatives_available: int # Count of comparable alternatives @property def risk_level(self) -> LockInRiskLevel: