Measure whether a skill, agent, or workflow actually changes AI agent behavior. Use when shipping a new skill, after modifying one, or comparing alternative approaches โ the eval gate behind the Iron Law.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Measure whether a skill, agent, or workflow actually changes AI agent behavior. Use when shipping a new skill, after modifying one, or comparing alternative approaches โ the eval gate behind the Iron Law.
category
meta
status
promoted
arc-evaluating
Measure whether skills, agents, and workflows actually change AI agent behavior. Define scenarios, prepare environments, run trials, grade results, track regressions.
Key distinction: You are evaluating AI agents (LLM + tools), not just LLM text output. Agents use tools, read files, and search codebases โ your eval environment must account for this.
When to Use
Eval is required when:
Shipping a new skill or agent
Modifying an existing skill
Comparing alternative approaches or prompts
Not required when: the change has no behavioral footprint (reformatting, typos, metadata-only edits). When in doubt, run the eval โ it is cheaper than shipping a regression.
Three Eval Scopes
1. Skill Evals
Does skill X change agent behavior?
Run scenario WITHOUT the skill (baseline)
Run scenario WITH the skill (treatment)
Compare outputs using grader
Measure: delta (improvement between baseline and treatment)
2. Agent Evals
Does agent Y produce correct output?
Run agent with a defined scenario
Grade output against acceptance criteria
Measure: pass@k (reliability across k trials)
3. Workflow Evals
Does the full toolkit system improve agent outcomes?
Baseline: Bare agent โ no plugins, no MCP, no skills/hooks
Treatment: Agent with full toolkit active (plugins, MCP, skills, hooks)
Same prompt, only the environment varies (skill evals vary the prompt instead)
Measure: delta, pass^k for critical paths
Scope Alignment (MANDATORY)
Before designing any scenario, confirm scope:
What is the eval target? (skill, agent, hook, pipeline)
What question are you answering? (match to table below)
What Claude behavior would change? If the answer is only side-effect artifacts (files, logs, counters) โ the eval harness is the wrong tool. Use unit tests.
Question
Scope
What Varies
Primary Signal
Does this instruction change agent behavior?
skill
Skill present vs absent
delta
Can this agent complete the task correctly?
agent
Trial-to-trial execution
pass@k, pass^k
Does the toolkit improve outcomes?
workflow
Bare agent vs full toolkit
delta, pass^k
Does this component work correctly?
none
N/A
Use unit/integration tests
Do NOT proceed to scenario design until you can answer question 2 in one sentence.
The Process
1. Preflight โ validate scenario is still discriminative
2. Define eval โ scenario + assertions + grader type
3. Prepare env โ setup the trial environment (files, tools, context)
4. Run eval โ spawn agent with scenario, capture transcript
5. Grade eval โ code grader, model grader, or human grader
6. Track resultsโ pass@k metric over time (JSONL)
7. Report โ SHIP / NEEDS WORK / BLOCKED / INSUFFICIENT_DATA
Before writing assertions, complete this checklist:
Can I name the specific Claude behavior this scenario tests? (If "file exists" or "no errors" โ you're testing infrastructure)
Would my assertions fail if I disabled the component under test? (If no โ no discriminative power)
Can I describe why baseline will fail? (If no โ scenario isn't discriminative)
Does each assertion use the right grader for its nature? (Code for facts, model for judgment)
Is output format small enough for consistent grading? (Prefer short structured artifacts)
Scenario validity rules:
Scenario files are single-condition. Do not put separate baseline and treatment sections into one scenario file. arc eval ab owns the A/B loop โ it runs the same single-condition scenario twice.
One behavior per scenario โ isolate one behavior so lift is attributable to one instruction
Include a trap or bait โ without a discriminative trap you're measuring generic competence, not skill adherence
Make ground truth defensible โ assertions must be supportable from provided context, not hidden conventions
Prefer 3-5 narrow scenarios over one overloaded scenario
See references/grading-and-execution.md for environment setup, trial execution, isolation mechanics, and result tracking. See references/cli-and-metrics.md for CLI commands, metrics, and the scenario template.
Grader Selection
Three graders: code (deterministic checks), model (intent/quality/reasoning), human (audience-dependent taste or domain expertise). Match grader to assertion nature โ not convenience. For discipline-skill compliance, agents/skill-grader.md also extracts and classifies rationalizations.
Grader selection principle: Structured output (JSON, typed fields) does not make semantic quality deterministic. An agent can return valid JSON while producing poor analysis. Code-grade structure; model-grade quality.
Model/human grader calibration: One vague model-grader preference is not release evidence. For semantic release claims, use a task-derived rubric with anchors, repeated trials, CI/variance/agreement checks, and blind comparison, human spot-check, or independent adjudication. Treat model-grader output as noisy semantic evidence, not deterministic proof.
Deterministic proxy warning: Keyword, regex, and JSON-schema checks cover facts/fields, not critique quality. If a proxy can pass a shallow or adversarial answer, tighten it with negative fixtures/traps or model/human-grade the quality claim.
Step 6: Report
Report behavior separately from operational cost. A treatment can be correct but slower, more verbose, or pricier. Preserve duration/token/cost deltas when available, and do not hide operational regressions behind a passing behavioral verdict.
k < 5 โ CI95 cannot be computed. Run more trials.
Full verdict semantics in references/verdict-policy.md.
Rationalization Table
When pressure builds to skip or shortcut eval, these rationalizations surface. Each is a blocker in disguise.
Excuse
Reality
"This change is too small to eval"
Size does not predict behavioral impact. A one-line prompt change can flip a verdict. Run eval โ it takes minutes.
"Time pressure, ship now and eval later"
Eval done after shipping is a postmortem, not a gate. Ship with evidence or do not ship.
"Preflight blocks โ I'll skip it just this once"
Preflight blocked because the scenario is no longer discriminative. Bypassing it means you cannot measure anything. Redesign the scenario.
"k=4 is close enough to 5"
The CI95 requires k โฅ 5 to be statistically meaningful. k=4 produces INSUFFICIENT_DATA. Run one more trial.
"INSUFFICIENT_DATA is advisory โ I'll ship anyway"
INSUFFICIENT_DATA means you have no valid statistical basis for a verdict. Shipping on INSUFFICIENT_DATA is shipping blind.
"The grader raised weak_assertions but the pass rate is fine"
weak_assertions signal the assertions are not testing the right thing. A passing score on a poorly designed assertion proves nothing. Redesign the assertion.
REQUIRED BACKGROUND: references/audit-workflow.md โ how promotion and retirement arbitration works for discovered_claims and weak_assertions.
Red Flags
Every listed thought means stop, re-read the skill, do not proceed.
"I already manually tested, eval is redundant" โ Manual testing measures your confidence, not the agent's behavioral reliability. Eval measures whether the skill systematically changes agent behavior across trials.
"This is docs-only, no eval needed" โ Docs changes that alter skill instructions change agent behavior by definition. If you changed what the agent reads, you changed what the agent does.
"The INSUFFICIENT_DATA banner is just a warning" โ INSUFFICIENT_DATA is a hard gate, not a warning. It means you have no statistical verdict. Shipping under INSUFFICIENT_DATA is shipping without evidence.
"I can promote the discovered claim on my own without audit" โ Promotion requires human arbitration to ensure the claim is generalizable and non-redundant. Bypassing audit corrupts the canonical skill body.
"The blind comparator disagreed but assertions passed so it's fine" โ The blind comparator is an independent signal. Disagreement between the comparator and assertion scores indicates one of them is poorly calibrated. Investigate before shipping.
"Preflight is new, I'll skip it this time and backfill later" โ Preflight is a gate, not a recommendation. Running trials on a scenario that fails preflight produces results you cannot trust. There is no backfill โ run preflight first.
Common Mistakes
Top mistakes that waste the most eval runs. Full catalog in references/common-mistakes-catalog.md.
Mistake
What Happens
Fix
Scenario before question
Mixing adherence, correctness, and toolkit effects in one noisy test
State the question first: behavior change, task outcome, or toolkit effect
Baseline already near ceiling
Both conditions pass, delta stays tiny
Run 2-3 pilot trials first; if baseline exceeds ~0.8, redesign
Prompt leaks the repair pattern
Baseline follows the template and scores high without the skill
Remove explicit grader split or named repair structure from the prompt
Using --skill-file for workflow eval
Varies the prompt instead of the environment
Workflow A/B varies the environment โ use eval ab <name> without --skill-file
Workflow eval with no plugins installed
Baseline and treatment are identical, delta is always 0
Ensure toolkit plugin is installed: claude plugin list should show active plugins
Integration
Design the target with arc-brainstorming; define success with arc-planning. Results inform whether to SHIP or iterate; track benchmarks over time in evals/benchmarks/latest.json.
Numeric vs qualitative analysis: Numeric comparison (delta, CI, verdict) is programmatic โ the harness computes it. The eval-analyzer agent adds qualitative analysis for model/human-graded A/B results; it does not replace the programmatic verdict.
Additional depth loads on demand: references/eval-schemas.md (JSON schemas for evals.json, grading/trial-N.json, comparison.json) and references/common-mistakes-catalog.md (full 23-entry catalog).
