| name | agent-evaluation |
| description | Design reproducible evaluations for AI agents with representative task sets, explicit rubrics, appropriate graders, baselines, regression gates, and failure analysis. Use when defining agent quality, comparing prompts or models, validating a release, measuring tool-use reliability, investigating regressions, or deciding whether an agent is ready for production. |
Agent Evaluation
Build evidence that can inform a release owner, not a showcase of favorable examples or a safety certification.
Use when
- Define quality before building or changing an agent.
- Compare prompts, models, tools, memory strategies, or orchestration patterns.
- Convert production failures into regression cases.
- Establish a repeatable release gate or human-review plan.
Inputs
Collect the agent objective, users, supported tasks, unacceptable outcomes, current baseline, execution environment, available traces, and evaluation budget. State assumptions when an input is unavailable.
Output contract
Produce:
- An evaluation brief with scope, risks, hypotheses, and frozen system versions.
- A dataset manifest with provenance, categories, splits, and contamination controls.
- A scoring specification with rubrics, graders, thresholds, and tie-breaking rules.
- Reproducible run settings, aggregate results, uncertainty, and baseline deltas.
- A failure taxonomy, representative cases, evidence limits, and a decision memo for the accountable release owner.
Workflow
- Define the unit under test and the decision the evaluation must support. Separate model quality from tool, retrieval, policy, and infrastructure failures.
- Convert user goals and risks into observable criteria. Include task success, safety, latency, cost, and escalation quality only when relevant.
- Build representative cases from real distributions where permitted. Add boundary, long-tail, malformed-input, tool-failure, and adversarial cases. Keep a holdout set isolated from prompt iteration.
