| name | arc-evaluating |
| description | Measure whether a skill, agent, or workflow actually changes AI agent behavior. Use when shipping a new skill, after modifying one, or comparing alternative approaches — the eval gate behind the Iron Law. |
| category | meta |
| status | promoted |
arc-evaluating
Measure whether skills, agents, and workflows actually change AI agent behavior. Define scenarios, prepare environments, run trials, grade results, track regressions.
Key distinction: You are evaluating AI agents (LLM + tools), not just LLM text output. Agents use tools, read files, and search codebases — your eval environment must account for this.
When to Use
Eval is required when:
- Shipping a new skill or agent
- Modifying an existing skill
- Comparing alternative approaches or prompts
Not required when: the change has no behavioral footprint (reformatting, typos, metadata-only edits). When in doubt, run the eval — it is cheaper than shipping a regression.
Three Eval Scopes
1. Skill Evals
Does skill X change agent behavior?
- Run scenario WITHOUT the skill (baseline)
- Run scenario WITH the skill (treatment)
- Compare outputs using grader
- Measure:
delta (improvement between baseline and treatment)
2. Agent Evals
Does agent Y produce correct output?
- Run agent with a defined scenario
- Grade output against acceptance criteria
- Measure:
pass@k (reliability across k trials)
3. Workflow Evals
Does the full toolkit system improve agent outcomes?
- Baseline: Bare agent — no plugins, no MCP, no skills/hooks
- Treatment: Agent with full toolkit active (plugins, MCP, skills, hooks)
- Same prompt, only the environment varies (skill evals vary the prompt instead)
- Measure:
delta, pass^k for critical paths
Scope Alignment (MANDATORY)
Before designing any scenario, confirm scope:
- What is the eval target? (skill, agent, hook, pipeline)
- What question are you answering? (match to table below)
- What Claude behavior would change? If the answer is only side-effect artifacts (files, logs, counters) — the eval harness is the wrong tool. Use unit tests.
| Question | Scope | What Varies | Primary Signal |
|---|
| Does this instruction change agent behavior? | skill | Skill present vs absent | delta |
| Can this agent complete the task correctly? | agent | Trial-to-trial execution | pass@k, pass^k |
| Does the toolkit improve outcomes? | workflow | Bare agent vs full toolkit | delta, pass^k |
| Does this component work correctly? | none | N/A | Use unit/integration tests |
Do NOT proceed to scenario design until you can answer question 2 in one sentence.
The Process
1. Preflight → validate scenario is still discriminative
2. Define eval → scenario + assertions + grader type
3. Prepare env → setup the trial environment (files, tools, context)
4. Run eval → spawn agent with scenario, capture transcript
5. Grade eval → code grader, model grader, or human grader
6. Track results→ pass@k metric over time (JSONL)
7. Report → SHIP / NEEDS WORK / BLOCKED / INSUFFICIENT_DATA
REQUIRED BACKGROUND: references/preflight.md — ceiling threshold (0.8), PASS/BLOCK semantics, scenario hash mechanics.
REQUIRED BACKGROUND: references/verdict-policy.md — full verdict enum (SHIP, NEEDS WORK, BLOCKED, IMPROVED, REGRESSED, INCONCLUSIVE, INSUFFICIENT_DATA), why k<5 triggers INSUFFICIENT_DATA, asymmetric delta thresholds.
Scenario Design Rules
Before writing assertions, complete this checklist:
- Can I name the specific Claude behavior this scenario tests? (If "file exists" or "no errors" — you're testing infrastructure)
- Would my assertions fail if I disabled the component under test? (If no — no discriminative power)
- Can I describe why baseline will fail? (If no — scenario isn't discriminative)
- Does each assertion use the right grader for its nature? (Code for facts, model for judgment)
- Is output format small enough for consistent grading? (Prefer short structured artifacts)
Scenario validity rules:
- Scenario files are single-condition. Do not put separate baseline and treatment sections into one scenario file.
arc eval ab owns the A/B loop — it runs the same single-condition scenario twice.
- One behavior per scenario — isolate one behavior so lift is attributable to one instruction
- Include a trap or bait — without a discriminative trap you're measuring generic competence, not skill adherence
- Make ground truth defensible — assertions must be supportable from provided context, not hidden conventions
- Prefer 3-5 narrow scenarios over one overloaded scenario
See references/grading-and-execution.md for environment setup, trial execution, isolation mechanics, and result tracking. See references/cli-and-metrics.md for CLI commands, metrics, and the scenario template.
Grader Selection
Three graders: code (deterministic checks), model (intent/quality/reasoning), human (audience-dependent taste or domain expertise). Match grader to assertion nature — not convenience. For discipline-skill compliance, agents/skill-grader.md also extracts and classifies rationalizations.
Grader selection principle: Structured output (JSON, typed fields) does not make semantic quality deterministic. An agent can return valid JSON while producing poor analysis. Code-grade structure; model-grade quality.
Model/human grader calibration: One vague model-grader preference is not release evidence. For semantic release claims, use a task-derived rubric with anchors, repeated trials, CI/variance/agreement checks, and blind comparison, human spot-check, or independent adjudication. Treat model-grader output as noisy semantic evidence, not deterministic proof.
Deterministic proxy warning: Keyword, regex, and JSON-schema checks cover facts/fields, not critique quality. If a proxy can pass a shallow or adversarial answer, tighten it with negative fixtures/traps or model/human-grade the quality claim.
Step 6: Report
Report behavior separately from operational cost. A treatment can be correct but slower, more verbose, or pricier. Preserve duration/token/cost deltas when available, and do not hide operational regressions behind a passing behavioral verdict.
| Verdict | Meaning |
|---|
| SHIP | Code-graded: pass rate = 100%. Model-graded: CI95 lower bound ≥ 0.8 |
| NEEDS WORK | 60% ≤ pass rate < SHIP threshold |
| BLOCKED | pass rate < 60% |
| INSUFFICIENT_DATA | k < 5 — CI95 cannot be computed. Run more trials. |
Full verdict semantics in references/verdict-policy.md.
Rationalization Table
When pressure builds to skip or shortcut eval, these rationalizations surface. Each is a blocker in disguise.
| Excuse | Reality |
|---|
| "This change is too small to eval" | Size does not predict behavioral impact. A one-line prompt change can flip a verdict. Run eval — it takes minutes. |
| "Time pressure, ship now and eval later" | Eval done after shipping is a postmortem, not a gate. Ship with evidence or do not ship. |
| "Preflight blocks — I'll skip it just this once" | Preflight blocked because the scenario is no longer discriminative. Bypassing it means you cannot measure anything. Redesign the scenario. |
| "k=4 is close enough to 5" | The CI95 requires k ≥ 5 to be statistically meaningful. k=4 produces INSUFFICIENT_DATA. Run one more trial. |
| "INSUFFICIENT_DATA is advisory — I'll ship anyway" | INSUFFICIENT_DATA means you have no valid statistical basis for a verdict. Shipping on INSUFFICIENT_DATA is shipping blind. |
| "The grader raised weak_assertions but the pass rate is fine" | weak_assertions signal the assertions are not testing the right thing. A passing score on a poorly designed assertion proves nothing. Redesign the assertion. |
REQUIRED BACKGROUND: references/audit-workflow.md — how promotion and retirement arbitration works for discovered_claims and weak_assertions.
Red Flags
Every listed thought means stop, re-read the skill, do not proceed.
- "I already manually tested, eval is redundant" — Manual testing measures your confidence, not the agent's behavioral reliability. Eval measures whether the skill systematically changes agent behavior across trials.
- "This is docs-only, no eval needed" — Docs changes that alter skill instructions change agent behavior by definition. If you changed what the agent reads, you changed what the agent does.
- "The INSUFFICIENT_DATA banner is just a warning" — INSUFFICIENT_DATA is a hard gate, not a warning. It means you have no statistical verdict. Shipping under INSUFFICIENT_DATA is shipping without evidence.
- "I can promote the discovered claim on my own without audit" — Promotion requires human arbitration to ensure the claim is generalizable and non-redundant. Bypassing audit corrupts the canonical skill body.
- "The blind comparator disagreed but assertions passed so it's fine" — The blind comparator is an independent signal. Disagreement between the comparator and assertion scores indicates one of them is poorly calibrated. Investigate before shipping.
- "Preflight is new, I'll skip it this time and backfill later" — Preflight is a gate, not a recommendation. Running trials on a scenario that fails preflight produces results you cannot trust. There is no backfill — run preflight first.
Common Mistakes
Top mistakes that waste the most eval runs. Full catalog in references/common-mistakes-catalog.md.
| Mistake | What Happens | Fix |
|---|
| Scenario before question | Mixing adherence, correctness, and toolkit effects in one noisy test | State the question first: behavior change, task outcome, or toolkit effect |
| Baseline already near ceiling | Both conditions pass, delta stays tiny | Run 2-3 pilot trials first; if baseline exceeds ~0.8, redesign |
| Prompt leaks the repair pattern | Baseline follows the template and scores high without the skill | Remove explicit grader split or named repair structure from the prompt |
Using --skill-file for workflow eval | Varies the prompt instead of the environment | Workflow A/B varies the environment — use eval ab <name> without --skill-file |
| Workflow eval with no plugins installed | Baseline and treatment are identical, delta is always 0 | Ensure toolkit plugin is installed: claude plugin list should show active plugins |
Integration
Design the target with arc-brainstorming; define success with arc-planning. Results inform whether to SHIP or iterate; track benchmarks over time in evals/benchmarks/latest.json.
Numeric vs qualitative analysis: Numeric comparison (delta, CI, verdict) is programmatic — the harness computes it. The eval-analyzer agent adds qualitative analysis for model/human-graded A/B results; it does not replace the programmatic verdict.
Additional depth loads on demand: references/eval-schemas.md (JSON schemas for evals.json, grading/trial-N.json, comparison.json) and references/common-mistakes-catalog.md (full 23-entry catalog).
