| name | eval-coach |
| description | Guide users through building comprehensive AI evaluation strategies using Evaluation-Driven Development (EDD) |
| metadata | {"version":"1.0.0","author":"AI Product Engineer School","triggers":["eval","evaluation","testing strategy","test cases","LLM testing"]} |
Eval Coach
An Agent Skill for designing comprehensive AI evaluation strategies using Evaluation-Driven Development (EDD).
Overview
Eval Coach guides you through a structured 5-step framework for evaluating LLM applications:
- Define Success - Map business goals to measurable metrics
- Design Dataset - Create diverse test cases (happy path, edge cases, adversarial)
- Select Methods - Choose Automated, LLM-as-Judge, or Human evaluation
- Plan Automation - Integrate evals into CI/CD
- Monitor Production - Track drift and collect feedback
When to Use This Skill
Invoke this skill when:
- Starting a new AI project and need an evaluation strategy
- Improving an existing agent's reliability
- Comparing different implementation approaches (e.g., LangGraph vs Deep Agents)
- Setting up CI/CD for AI products
- Debugging production quality issues
Evaluation Philosophy
The 50-40-10 Rule
- 50% Automated - Schema validation, keyword checks, latency ($0.00/run)
- 40% LLM-as-Judge - Semantic quality, relevance ($0.01-0.05/run)
- 10% Human - Subjective quality, edge cases ($5-50/run)
