skill-eval-builder

name	skill-eval-builder
description	Use when the user wants to measure or set up evals/checks for one of their skills — how fast it is, whether its output is valid, whether it fires when expected, or whether its opening classification/routing gate labels inputs correctly.

Skill Eval Builder

Overview

Set up a small, real eval for a skill: an evals/ folder next to it with a few real cases, a runnable script, and a scorecard. It measures what scripts can't pin down — does the skill fire when it should (and stay quiet when it shouldn't), is its output valid, is it within a time budget.

Core principle: an eval is a folder, not a framework. Keep it that small.

Inputs

• Target skill — path or name (the dir with its SKILL.md).
• A few real cases — should-fire and shouldn't-fire prompts + real inputs; help the user find them if needed.
• Which dimensions matter — default invocation + validation; add duration/others if they fit.

Steps

1. Read the target skill — its SKILL.md, references, and scripts, so you know what it does and what it produces.
2. Pick dimensions that fit it, from references/eval-dimensions.md. Workflow skill → invocation + duration; capability skill → validation-heavy.
3. Gather 3–5 real cases — should-fire and shouldn't-fire prompts plus real inputs. Always include at least one quiet case.
4. Define "good" per case — a fired_when side-effect check, a validate command, and a budget_s from a real baseline (see references/eval-anatomy.md).