| name | ai-vendor-evaluation |
| description | Comprehensive framework for evaluating AI vendors and solutions to avoid costly mistakes. Use this skill when assessing AI vendor proposals, conducting due diligence, evaluating contracts, comparing vendors, or making build-vs-buy decisions. Helps identify red flags, assess pricing models, evaluate technical capabilities, and conduct structured vendor comparisons. |
AI Vendor Evaluation
Version 1.0 | October 2025 | Based on $1.2M average AI spend analysis
Overview
This skill provides a systematic framework for evaluating AI vendors and solutions to avoid the costly mistakes that plague 95% of AI projects. Use when conducting vendor due diligence, evaluating proposals, negotiating contracts, or making strategic AI purchasing decisions.
Key capabilities:
- Structured evaluation criteria for AI vendors
- Red flag identification in proposals and demos
- Pricing model analysis and fair market rates
- Technical capability assessment
- Contract term evaluation
- Build vs buy decision framework
Quick Decision Tree
Start here to determine which references to read:
What stage are you in?
├─ Early exploration (multiple vendors being considered)
│ └─ Read: evaluation-criteria.md, use-case-fit.md
│ Use: scorecard-template.xlsx
│
├─ Evaluating specific proposal or demo
│ └─ Read: red-flags.md, technical-assessment.md
│ Check: pricing-models.md for pricing reasonableness
│
├─ Contract negotiation
│ └─ Read: contract-checklist.md, pricing-models.md
│ Reference: red-flags.md for problematic terms
│
├─ Build vs Buy decision
│ └─ Read: build-vs-buy.md, use-case-fit.md
│ Consider: Total cost of ownership from pricing-models.md
│
└─ Post-purchase review or audit
└─ Read: evaluation-criteria.md, technical-assessment.md
Assess: Whether vendor is delivering on promises
When to Use This Skill
Trigger scenarios:
- "Help me evaluate this AI vendor proposal"
- "What should I look for in AI vendor demos?"
- "Is this pricing reasonable for an AI solution?"
- "Should we build or buy this AI capability?"
- "What questions should I ask this AI vendor?"
- "Help me compare these AI vendors"
- "Review this AI contract for red flags"
- "Conduct due diligence on this AI company"
Core Evaluation Framework
Phase 1: Initial Screening
Goal: Eliminate obviously problematic vendors before deep evaluation
Key questions:
- Does the vendor have relevant domain experience?
- Are there verifiable customer references?
- Is the technology approach sound?
- Are pricing and terms transparent?
Read: references/red-flags.md for disqualifying signals
Read: references/use-case-fit.md for domain fit assessment
Phase 2: Deep Evaluation
Goal: Assess vendor capabilities systematically across all dimensions
Evaluation dimensions:
- Technical capability - Can they actually deliver?
- Business viability - Will they still exist in 2 years?
- Pricing fairness - Are costs reasonable for value delivered?
- Implementation risk - How likely is successful deployment?
- Contract terms - Are legal terms acceptable?
Read: references/evaluation-criteria.md for comprehensive framework
Read: references/technical-assessment.md for technical evaluation
Read: references/pricing-models.md for pricing analysis
Use: assets/scorecard-template.xlsx to score vendors systematically
Phase 3: Contract Negotiation
Goal: Secure favorable terms and avoid costly traps
Critical areas:
- Performance guarantees and SLAs
- Data ownership and usage rights
- Pricing structure and escalation terms
- Exit clauses and data portability
- Liability and indemnification
Read: references/contract-checklist.md for essential terms
Reference: references/red-flags.md for problematic contract patterns
Common Vendor Patterns
The Overpromiser
Characteristics: Claims to solve everything, vague on technical details, aggressive sales tactics
Red flag: "Our AI can handle any use case"
Response: Demand specific technical explanations and verifiable references
The Feature Dumper
Characteristics: Long feature lists, complex pricing, unclear core value proposition
Red flag: Can't explain what problem they actually solve
Response: Force clarity on primary use case and success metrics
The Consultant in Disguise
Characteristics: Software license + mandatory professional services
Red flag: Professional services cost more than software
Response: Assess true cost of ownership, consider if you're buying software or consulting
The Model Wrapper
Characteristics: Thin layer over OpenAI/Anthropic APIs with high markup
Red flag: No proprietary technology, just API access + UI
Response: Calculate cost of building similar solution in-house
Full pattern library: See references/red-flags.md
Build vs Buy Decision Framework
When to read this section: Before committing to vendor evaluation, determine if building in-house is better option.
Key factors:
- Capability availability - Does suitable vendor solution exist?
- Time to value - Buy: weeks-months, Build: months-years
- Total cost - Consider 3-year TCO for both options
- Strategic importance - Core competency? Build. Commodity? Buy.
- Team capability - Do you have talent to build and maintain?
Read: references/build-vs-buy.md for detailed decision framework
Using the Scorecard Template
The vendor scorecard enables structured comparison across vendors.
To use:
- Open
assets/scorecard-template.xlsx
- List vendors to compare (up to 5)
- Score each vendor on evaluation criteria (1-5 scale)
- Review weighted scores and vendor comparison chart
- Document decision rationale
Customization: Adjust weights based on priorities for your specific use case.
Reference Documents
references/evaluation-criteria.md
Comprehensive scoring framework across all vendor evaluation dimensions. Includes specific questions to ask, what constitutes good/bad answers, and how to weight criteria for different use cases.
Use when: Conducting systematic vendor evaluation
references/red-flags.md
Catalog of warning signs indicating problematic vendors. Organized by category: technical red flags, business red flags, pricing red flags, contract red flags, and behavioral red flags.
Use when: Initial vendor screening or reviewing proposals
references/pricing-models.md
Guide to AI vendor pricing models (per-seat, usage-based, platform fees, etc.), fair market rates, what drives costs, and how to negotiate. Includes pricing red flags and total cost of ownership analysis.
Use when: Evaluating vendor pricing or negotiating contracts
references/technical-assessment.md
Framework for assessing technical capabilities: architecture review, model evaluation, integration complexity, scalability, security, and data handling. Includes specific technical questions to ask.
Use when: Deep technical evaluation of vendor capabilities
references/contract-checklist.md
Essential contract terms for AI vendor agreements: performance guarantees, data rights, pricing protection, exit terms, liability, and support commitments. Includes negotiation guidance.
Use when: Contract review or negotiation
references/use-case-fit.md
Framework for assessing whether vendor solution actually fits your use case. Includes questions to ask yourself, questions to ask vendor, and warning signs of poor fit.
Use when: Initial vendor screening or use case definition
references/build-vs-buy.md
Decision framework for whether to build AI capability in-house vs purchasing vendor solution. Includes total cost analysis, capability assessment, and strategic considerations.
Use when: Before committing to vendor evaluation process
Assets
assets/scorecard-template.xlsx
Structured spreadsheet for vendor comparison with:
- Evaluation criteria organized by category
- Scoring system (1-5 scale) with descriptions
- Weighted scoring based on priorities
- Vendor comparison charts
- Decision documentation section
Customize: Adjust criteria weights and add company-specific requirements
