| name | product-ai-evals |
| description | Design an AI-product evaluation system connecting task quality, user outcomes, failure taxonomies, datasets, human judgment, automated checks, release thresholds, and production monitoring. Use when defining AI quality, comparing models or prompts, deciding launch readiness, or diagnosing regressions. |
Product AI Evals
Evaluate the end-to-end product task, not only a model metric. Connect offline tests, human judgment, controlled exposure, and production outcomes.
Inputs
- User task, workflow, and intended outcome
- Output types and acceptable variation
- Failure consequences and affected segments
- Current system, baseline, and alternatives
- Representative data and privacy constraints
- Cost, latency, and review budgets
Workflow
- Decompose the workflow into observable task steps and outcomes.
- Create a failure taxonomy covering incorrectness, omission, unsupported content, refusal, bias, unsafe action, poor recovery, latency, cost, and user misunderstanding.
- Build a representative evaluation set from real task distributions, difficult cases, high-impact cases, and adversarial cases. Track provenance and consent.
- Define rubrics with examples of pass, partial, and fail; use pairwise comparison when absolute scoring is unreliable.
- Combine deterministic checks, model or heuristic graders, expert review, and user outcome measures according to consequence.
- Establish baseline, candidate, segment slices, confidence intervals where appropriate, and regression budget.
- Set release thresholds and non-compensable safety or policy gates.
- Validate grader agreement and inspect disagreements; do not treat an automated judge as ground truth.
- Connect offline results to staged production monitoring and feedback.
Output contract
Return task definition, failure taxonomy, dataset plan, rubric, evaluator mix, metric definitions, segment slices, baselines, thresholds, release rule, monitoring links, and unresolved limitations.
Quality gate
- Evaluation examples match real usage and consequential edge cases.
- Product outcomes and model behavior are both measured.
- Aggregate scores cannot hide critical failure categories.
- Grader validity and human disagreement are monitored.
- Cost and latency are included in quality.
Avoid
- One generic “accuracy” score
- Testing only happy-path prompts
- Reusing production feedback without provenance review
- Optimizing to a judge without spot checks
- Launching because the average improved while a critical segment regressed
Source grounding
Operational synthesis informed by success-metric, trade-off, validation, and launch practices in Building AI-Powered Products and evaluation and production-system principles in Designing Machine Learning Systems.
