| name | eval-runner |
| description | Create and run agent evals to verify AI output quality. Use when the user asks to test agent outputs, create eval cases, or benchmark skill performance. |
| category | evaluation |
| version | v1.0 |
| inputs | ["skill name","test prompts","expected outputs"] |
| outputs | ["Eval JSON files","Eval results","Pass/fail report"] |
Eval Runner Skill
Purpose
Define, run, and report on evaluation cases for agent skills to confirm output quality and correctness.
When To Use
Use this skill when the user asks to:
- Create eval cases for a skill
- Test whether an agent produces correct output
- Benchmark skill performance
- Validate agent routing behavior
Inputs
- Skill name
- Test prompts (natural language)
- Expected outputs (structured description)
- Relevant files for context
Workflow
- Identify the skill or agent being evaluated
- Write 2–5 eval cases covering core use cases and edge cases
- Define expected output for each case (structured, not exact match)
- Save as JSON in
evals/
- Run eval prompts and compare outputs to expected
- Report pass/fail with notes
Output Format
{
"skill_name": "[name]",
"evals": [
{
"id": 1,
"name": "[eval-name]",
"prompt": "[test prompt]",
"expected_output": "[description of correct output]",
"files": []
}
]
}
Quality Checklist
References
