skill-eval-scaffolder

name	skill-eval-scaffolder
description	Scaffold evals/evals.json manifests for repo skills with baseline cases. Use when adding behavioral eval coverage. NOT for live LLM eval execution.
user-invocable	true
argument-hint	<skill-name> [--apply]
license	MIT
metadata	{"author":"wyattowalsh","version":"1.0.0"}

Skill Eval Scaffolder

Create minimal, schema-valid eval manifests for repository skills.

Scope: Eval scaffolding only. Do not run live behavioral evals unless the maintainer explicitly approves.

Dispatch

$ARGUMENTS	Action
<name>	Preview scaffold for skills/<name>/evals/evals.json
<name> --apply	Write scaffold when file is missing or user approved overwrite
check <name>	Validate an existing eval manifest
Empty	Show usage gallery and manifest schema summary

Workflow

1. Confirm skills/<name>/SKILL.md exists.
2. Run the bundled scaffold script:

uv run python skills/skill-eval-scaffolder/scripts/scaffold_evals.py <name>

3. Add explicit, implicit, and negative-control cases before claiming adequate coverage.
4. Validate with the bundled check script.

Manifest Shape

Each evals/evals.json uses:

• skill_name — kebab-case directory name
• evals — non-empty list of cases with id, prompt, expected_output, optional files, assertions

Validation Contract

uv run python skills/skill-eval-scaffolder/scripts/check.py

Critical Rules

1. Never overwrite an existing eval manifest without explicit user approval.
2. Include at least one explicit-invocation case in new scaffolds.
3. Run validate_evals.py before declaring the scaffold complete.
4. Route live eval execution to skill-creator eval mode or maintainer CI workflows.
5. Keep prompts grounded in the skill dispatch table and NOT-for boundaries.