run-evals

name	run-evals
description	Run the skill eval pipeline to validate and test COO skills. Use when checking skill quality, after modifying a SKILL.md, or when asked to run evals, test skills, or validate skills.
allowed-tools	Bash Read Glob

Run Skill Evals

Execute the three-layer skill evaluation pipeline for COO skills.

Quick Start

Run all layers:

/root/projects/COO/evals/skill-evals/run-skill-evals.sh

Eval Layers

Layer 1: Structural Validation (validate)

Checks every SKILL.md against the agentskills.io spec:

• Required frontmatter fields (name, description)
• Name format (kebab-case, max 64 chars, matches directory)
• Description length and content
• Body line count (recommended < 500)
• Enterprise-specific conventions

/root/projects/COO/evals/skill-evals/run-skill-evals.sh validate

Layer 2: Trigger Accuracy (trigger)

Tests whether skills activate for the right prompts. Requires claude CLI. Uses eval cases from evals/skill-evals/trigger-evals.json.

For interactive trigger testing:

1. Read the trigger eval cases from evals/skill-evals/trigger-evals.json
2. For each case, reason about whether the skill's description would cause activation
3. Flag any description improvements needed

Layer 3: Behavioral Grading (grade)

Grades a captured skill execution transcript against assertions.

1. Execute a skill (e.g., /handoff, /health-check)
2. Capture the full output to a file
3. Grade it:

/root/projects/COO/evals/skill-evals/run-skill-evals.sh grade /path/to/transcript.txt

Assertions are auto-graded where possible (command presence, pattern matching). Assertions requiring judgment are flagged for human review.

Eval Files

File	Skill	Cases
handoff-evals.json	handoff	4 cases: basic entry, fresh file, with/without files changed
daily-housekeeping-evals.json	daily-housekeeping	4 cases: archive, idempotency, missing file, legacy cleanup
health-check-evals.json	health-check	3 cases: basic check, gtuser pattern, not-update boundary

Adding Evals for a New Skill

Create <skill-name>-evals.json in evals/skill-evals/ following this format:

{
  "skill_name": "my-skill",
  "skill_path": ".claude/skills/my-skill",
  "description": "What these evals test",
  "format_version": "1.0",
  "evals": [
    {
      "id": 1,
      "name": "descriptive-test-name",
      "prompt": "The user prompt that invokes the skill",
      "expected_output": "Human-readable description of correct behavior",
      "assertions": [
        "Specific checkable claim about the output",
        "Another verifiable assertion"
      ]
    }
  ]
}

Results

All results are written to evals/skill-evals/results/ with timestamps. Run report to see a summary of the latest results:

/root/projects/COO/evals/skill-evals/run-skill-evals.sh report