| name | run-evals |
| description | Run the skill eval pipeline to validate and test COO skills. Use when checking skill quality, after modifying a SKILL.md, or when asked to run evals, test skills, or validate skills. |
| allowed-tools | Bash Read Glob |
Run Skill Evals
Execute the three-layer skill evaluation pipeline for COO skills.
Quick Start
Run all layers:
/root/projects/COO/evals/skill-evals/run-skill-evals.sh
Eval Layers
Layer 1: Structural Validation (validate)
Checks every SKILL.md against the agentskills.io spec:
- Required frontmatter fields (name, description)
- Name format (kebab-case, max 64 chars, matches directory)
- Description length and content
- Body line count (recommended < 500)
- Enterprise-specific conventions
/root/projects/COO/evals/skill-evals/run-skill-evals.sh validate
Layer 2: Trigger Accuracy (trigger)
Tests whether skills activate for the right prompts. Requires claude CLI.
Uses eval cases from evals/skill-evals/trigger-evals.json.
For interactive trigger testing:
- Read the trigger eval cases from
evals/skill-evals/trigger-evals.json
- For each case, reason about whether the skill's description would cause activation
- Flag any description improvements needed
Layer 3: Behavioral Grading (grade)
Grades a captured skill execution transcript against assertions.
- Execute a skill (e.g.,
/handoff, /health-check)
- Capture the full output to a file
- Grade it:
/root/projects/COO/evals/skill-evals/run-skill-evals.sh grade /path/to/transcript.txt
Assertions are auto-graded where possible (command presence, pattern matching).
Assertions requiring judgment are flagged for human review.
Eval Files
| File | Skill | Cases |
|---|
handoff-evals.json | handoff | 4 cases: basic entry, fresh file, with/without files changed |
daily-housekeeping-evals.json | daily-housekeeping | 4 cases: archive, idempotency, missing file, legacy cleanup |
health-check-evals.json | health-check | 3 cases: basic check, gtuser pattern, not-update boundary |
Adding Evals for a New Skill
Create <skill-name>-evals.json in evals/skill-evals/ following this format:
{
"skill_name": "my-skill",
"skill_path": ".claude/skills/my-skill",
"description": "What these evals test",
"format_version": "1.0",
"evals": [
{
"id": 1,
"name": "descriptive-test-name",
"prompt": "The user prompt that invokes the skill",
"expected_output": "Human-readable description of correct behavior",
"assertions": [
"Specific checkable claim about the output",
"Another verifiable assertion"
]
}
]
}
Results
All results are written to evals/skill-evals/results/ with timestamps.
Run report to see a summary of the latest results:
/root/projects/COO/evals/skill-evals/run-skill-evals.sh report
