claude-code-evals

name

claude-code-evals

description

MANDATORY load before writing or editing any eval scenario (scenario.yaml), check (checks.yaml), or evals.yaml — wherever the eval suite lives. Evaluate Claude Code configurations (skills, CLAUDE.md, hooks, MCP servers, settings, sub-agents) using the scuttlerun/pincenez/craboodle eval pipeline. TRIGGER when: authoring or modifying scenario.yaml or checks.yaml files; designing checks; interpreting eval results; benchmarking configurations; testing CLAUDE.md effectiveness; verifying hooks work; regression testing configs; building or extending an eval suite. DO NOT skip because 'I'll consult check-design.md directly' or 'one quick check edit' — load first, the schema and anti-pattern rules are cheap to surface.

Claude Code Evals

Evaluate Claude Code configurations with behavioral evidence.

When to Use

• Verifying a skill, CLAUDE.md instruction, hook, or MCP server works as intended
• Regression testing after changes to any config component
• Iterating on scenarios and check quality
• Building confidence before sharing or publishing a configuration
• Deciding whether a configuration change actually improved behavior

The Eval Mindset

Evaluation replaces intuition with behavioral evidence. Instead of reading your configuration and guessing whether it works, you run Claude against realistic tasks and measure outcomes.

The key question: "Would Claude behave the same way without this configuration?" If yes, your configuration may not be adding value — or your checks may not be targeting what the configuration adds.

When to evaluate:

• After any behavioral change to a configuration
• Before sharing or publishing configs for others to use
• When debugging inconsistent behavior — evals reveal whether the problem is the config or the task

When NOT to evaluate:

• Trivial formatting-only changes (whitespace, comments, reordering)
• Configs with no behavioral expectations (e.g., editor settings)
• During initial exploratory drafting — write first, evaluate when the config stabilizes

Evaluation is probabilistic. LLM behavior is non-deterministic. Evals run multiple repetitions and report pass rates, not binary pass/fail. A pass rate of 0.8 means the configuration works reliably, not that it works every time. Lint is non-deterministic too: each check is a single un-seeded LLM judgment, so re-running craboodle lint may add or drop individual findings (see pincenez lint --help). Treat lint findings as advisory — judge each on its merits rather than expecting an identical issue count on every run.

The Three Tools

craboodle (orchestrator)
  ├── scuttlerun (session driver) ── runs Claude, produces transcript
  └── pincenez (grader) ────────── grades transcript, produces verdict

• scuttlerun runs a headless Claude session with a synthetic user (LLM oracle), producing a full transcript. It handles multi-turn interactions and project scaffolding.
• pincenez grades one transcript against a checks file. Each check gets an independent LLM call — binary pass/fail with evidence. No cross-contamination between verdicts. The CLI has two modes: pincenez checks.yaml output grades (the unmarked default action, invoked positionally by craboodle), and pincenez lint checks.yaml lints checks for quality anti-patterns before grading.
• craboodle discovers scenarios, runs each through scuttlerun N times, grades each run with pincenez, averages results, and streams YAML output.

Run <tool> --help for CLI flags, YAML schemas, and field references.

Quick Start

Evaluate whether a CLAUDE.md instruction changes behavior:

1. Scaffold the eval root:

craboodle init my-evals/

This creates my-evals/evals.yaml — a single, fully-commented config holding both pipeline knobs (top level) and a scuttlerun base config (under scenarios.base). The eval root can be a skill directory (next to SKILL.md), a plugin root (next to .claude-plugin/plugin.json), or any generic directory.

2. Edit my-evals/evals.yaml:

version: "1"
min_pass_rate: 0.8

scenarios:
  base:
    model: claude-sonnet-4-6
    additional_tools:
      - TaskCreate
      - TaskUpdate
    project:
      claude_md: |
        Always write tests before production code. Use test-driven development.

additional_tools: vs tools: — additional_tools: extends scuttlerun's defaults: entries are appended after the defaults, with duplicates removed on a first-wins basis so original ordering is preserved. tools: replaces the defaults entirely. Use additional_tools: when you want defaults plus extras (e.g. TaskCreate, TaskUpdate, Agent); use tools: when you need an exact set (e.g. dropping Skill if no skills are under test, or AskUserQuestion if no multi-turn interaction). Run scuttlerun --help for the default tools list. An exact tools: set must still include every tool the config-under-test mandates — never trim below that to bound cost (see references/config-type-patterns.md § Skills for the failure mode).

3. Review lint rules (before writing checks, learn what lint looks for):

pincenez lint --help

4. Write my-evals/evals/tdd-instruction/scenario.yaml (per-scenario scuttlerun config):

prompt: |
  Write a function called isPrime that checks if a number is prime.
  Save it to prime.js.

5. Write my-evals/evals/tdd-instruction/checks.yaml (pincenez config):

checks:
  - test-before-code:
      check: "A test file was written before or alongside the production code"
      note: "Look for a test file created via the Write tool"
  - function-exists:
      check: "The function isPrime exists in prime.js"
  - tests-validate:
      check: "At least one test case validates prime number behavior"

6. Lint checks (catches anti-patterns before spending money):

craboodle lint my-evals/

7. Run:

craboodle run my-evals/

8. Interpret results — see references/results-interpretation.md.

What Can Be Evaluated

Any Claude Code configuration that changes agent behavior. Each scenario.yaml contains scuttlerun fields that define the configuration under test:

Config Type	Inject Via	What to Test
Skills	project.skills	Behavioral change with skill loaded
CLAUDE.md	project.claude_md	Instruction compliance, stickiness under pressure
Hooks/Settings	project.settings	Hook fires, affects behavior
MCP Servers	sdk.mcp_servers	Tool discovery and usage
Sub-agents	tools: [Agent] + sdk.agents	Delegation and coordination
Workflow-delegating skills	project.skills + Workflow tool	Skill delegates to a background workflow, waits, reads the artifact, presents without fabricating
Model/Effort	model, effort	Quality at different cost points
Plugins	project.plugins	Plugin components compose; skills + hooks + agents + MCP load and interact as one unit
Bundled Combos	Multiple fields	Combined config stack works as a unit

See references/config-type-patterns.md for scenario examples and design guidance for each type. For a guided bootstrap of plugin-level eval coverage, see plugincraft's workflows/bootstrap-evals.md.

Suite Lifecycle

An eval suite evolves alongside the configuration it tests:

• Growing the suite — Add scenarios when you add new behavior, discover edge cases, or find untested dimensions
• Improving checks — Iterate on check quality; fix always-passes anti-patterns, tighten vague checks, add missing note: fields
• Running after changes — Run your suite after config changes to catch regressions. Review any scenarios with degraded pass rates
• Retiring scenarios — Remove stale scenarios that no longer test meaningful behavior (e.g., config changed, scenario tests removed functionality)

See references/scenario-design.md for scenario structure and suite evolution guidance.

Reference Files

File	Purpose
references/check-design.md	Check patterns, anti-patterns, quality criteria
references/scenario-design.md	Scenario structure, suite evolution
references/config-type-patterns.md	Per-config-type eval guidance with examples
references/results-interpretation.md	Reading results, decision framework, iteration
references/config-precedence.md	Full config precedence chain across all three tools

Dependencies

• craboodle — eval orchestrator (craboodle --help)
• scuttlerun — session driver (scuttlerun --help)
• pincenez — checks grader (pincenez --help)
• skillcraft (sibling skill in this plugin) — for skill-specific eval patterns (pressure testing by skill type, trigger testing, worked examples per skill category)