| name | llm-evaluation |
| description | Design and run an evaluation for an AI or agent feature using representative tasks, explicit success criteria, failure taxonomy, baselines, and reproducible evidence. |
LLM Evaluation
Use before relying on a model, prompt, retrieval flow, or agent behavior in a consequential product workflow.
Procedure
- Define the behavior being evaluated and the real user/task distribution it must serve.
- Build representative test cases including normal, difficult, adversarial, ambiguous, and known-failure examples.
- Define scoring criteria before running the evaluation. Prefer observable task success and structured rubrics over vibes.
- Establish a meaningful baseline: previous prompt/model, simpler method, human result, or deterministic system where appropriate.
- Run with fixed configuration and record model, prompt/version, tools, retrieval inputs, temperature/reasoning settings, and relevant environment.
- Classify failures by cause: instruction following, reasoning, retrieval, tool use, format, hallucination, safety, latency, or cost.
- Compare changes on the same evaluation set and inspect regressions, not just average score.
Quality gate
A model change is better only when evidence shows improvement on the intended task without unacceptable regression, cost, latency, or safety tradeoffs.
