eval-harness

name	eval-harness
description	Eval-driven development (EDD) framework — define pass/fail criteria before implementing a feature, run capability and regression evals with pass@k metrics, and produce eval reports using code, model, or human graders

Eval Harness Skill

A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles.

When to Activate

• Setting up eval-driven development (EDD) for AI-assisted workflows
• Defining pass/fail criteria for Claude Code task completion
• Measuring agent reliability with pass@k metrics
• Creating regression test suites for prompt or agent changes
• Benchmarking agent performance across model versions

Philosophy

Eval-Driven Development treats evals as the "unit tests of AI development":

• Define expected behavior BEFORE implementation
• Run evals continuously during development
• Track regressions with each change
• Use pass@k metrics for reliability measurement

Eval Types

Capability Evals

Test if Claude can do something it couldn't before:

[CAPABILITY EVAL: feature-name]
Task: Description of what Claude should accomplish
Success Criteria:
  - [ ] Criterion 1
  - [ ] Criterion 2
  - [ ] Criterion 3
Expected Output: Description of expected result

Regression Evals

Ensure changes don't break existing functionality:

[REGRESSION EVAL: feature-name]
Baseline: SHA or checkpoint name
Tests:
  - existing-test-1: PASS/FAIL
  - existing-test-2: PASS/FAIL
  - existing-test-3: PASS/FAIL
Result: X/Y passed (previously Y/Y)

Grader Types

1. Code-Based Grader

Deterministic checks using code:

# Check if file contains expected pattern
grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL"

# Check if tests pass
npm test -- --testPathPattern="auth" && echo "PASS" || echo "FAIL"

# Check if build succeeds
npm run build && echo "PASS" || echo "FAIL"

2. Model-Based Grader

Use Claude to evaluate open-ended outputs:

[MODEL GRADER PROMPT]
Evaluate the following code change:
1. Does it solve the stated problem?
2. Is it well-structured?
3. Are edge cases handled?
4. Is error handling appropriate?

Score: 1-5 (1=poor, 5=excellent)
Reasoning: [explanation]

3. Human Grader

Flag for manual review:

[HUMAN REVIEW REQUIRED]
Change: Description of what changed
Reason: Why human review is needed
Risk Level: LOW/MEDIUM/HIGH

Metrics

pass@k

"At least one success in k attempts"

• pass@1: First attempt success rate
• pass@3: Success within 3 attempts
• Typical target: pass@3 > 90%

pass^k

"All k trials succeed"

• Higher bar for reliability
• pass^3: 3 consecutive successes
• Use for critical paths

Eval Workflow

1. Define (Before Coding)

## EVAL DEFINITION: feature-xyz

### Capability Evals
1. Can create new user account
2. Can validate email format
3. Can hash password securely

### Regression Evals
1. Existing login still works
2. Session management unchanged
3. Logout flow intact

### Success Metrics
- pass@3 > 90% for capability evals
- pass^3 = 100% for regression evals

2. Implement

Write code to pass the defined evals.

3. Evaluate

# Run capability evals
[Run each capability eval, record PASS/FAIL]

# Run regression evals
npm test -- --testPathPattern="existing"

# Generate report

4. Report

EVAL REPORT: feature-xyz
========================

Capability Evals:
  create-user:     PASS (pass@1)
  validate-email:  PASS (pass@2)
  hash-password:   PASS (pass@1)
  Overall:         3/3 passed

Regression Evals:
  login-flow:      PASS
  session-mgmt:    PASS
  logout-flow:     PASS
  Overall:         3/3 passed

Metrics:
  pass@1: 67% (2/3)
  pass@3: 100% (3/3)

Status: READY FOR REVIEW

Conceptual Workflow Commands

These are conceptual stages, not implemented slash commands. In the Kadmon Harness the same flow is driven by /abra-kdabra (konstruct drafts the eval definition during planning, feniks executes it during red-green-refactor cycles) and closed by /chekpoint (kody reviews the final state). Use them as a mental model when narrating your progress:

• Pre-implementation (define) — write the eval definition file at .claude/evals/<feature>.md before touching any production code
• During implementation (check) — re-run the eval definition against the evolving code after each green step
• Post-implementation (report) — produce the final eval report as part of /chekpoint or PR description

Eval Storage

Store evals in project:

.claude/
  evals/
    feature-xyz.md      # Eval definition
    feature-xyz.log     # Eval run history
    baseline.json       # Regression baselines

Best Practices

1. Define evals BEFORE coding - Forces clear thinking about success criteria
2. Run evals frequently - Catch regressions early
3. Track pass@k over time - Monitor reliability trends
4. Use code graders when possible - Deterministic > probabilistic
5. Human review for security - Never fully automate security checks
6. Keep evals fast - Slow evals don't get run
7. Version evals with code - Evals are first-class artifacts

Example: Adding Authentication

## EVAL: add-authentication

### Phase 1: Define (10 min)
Capability Evals:
- [ ] User can register with email/password
- [ ] User can login with valid credentials
- [ ] Invalid credentials rejected with proper error
- [ ] Sessions persist across page reloads
- [ ] Logout clears session

Regression Evals:
- [ ] Public routes still accessible
- [ ] API responses unchanged
- [ ] Database schema compatible

### Phase 2: Implement (varies)
[Write code]

### Phase 3: Evaluate
Run: /eval check add-authentication

### Phase 4: Report
EVAL REPORT: add-authentication
==============================
Capability: 5/5 passed (pass@3: 100%)
Regression: 3/3 passed (pass^3: 100%)
Status: SHIP IT

Product Evals (v1.8)

Use product evals when behavior quality cannot be captured by unit tests alone.

Grader Types

1. Code grader (deterministic assertions)
2. Rule grader (regex/schema constraints)
3. Model grader (LLM-as-judge rubric)
4. Human grader (manual adjudication for ambiguous outputs)

pass@k Guidance

• pass@1: direct reliability
• pass@3: practical reliability under controlled retries
• pass^3: stability test (all 3 runs must pass)

Recommended thresholds:

• Capability evals: pass@3 >= 0.90
• Regression evals: pass^3 = 1.00 for release-critical paths

Eval Anti-Patterns

• Overfitting prompts to known eval examples
• Measuring only happy-path outputs
• Ignoring cost and latency drift while chasing pass rates
• Allowing flaky graders in release gates

Minimal Eval Artifact Layout

• .claude/evals/<feature>.md definition
• .claude/evals/<feature>.log run history
• docs/releases/<version>/eval-summary.md release snapshot

Integration

• feniks agent (sonnet) — primary owner. feniks already enforces red-green-refactor at the unit level; this skill extends the same discipline to feature-level acceptance. When /abra-kdabra routes to feniks with needs_tdd: true, feniks writes capability + regression evals before any production code lands.
• konstruct agent (opus) — secondary. Before decomposing a feature into steps, konstruct reads this skill to express the feature's success in measurable pass/fail evals. The eval definition becomes the north star for the step breakdown.
• /abra-kdabra command — the chain arkitect → konstruct → feniks → kody is where EDD actually runs. konstruct drafts the evals as part of the plan; feniks implements them; kody reviews both production code and eval quality. This is the user-facing entry point for defining and running EDD evals on a feature.
• /chekpoint command — run evals as part of the verification loop before committing; treat failing capability or regression evals as BLOCK-level issues.

no_context Application

Eval definitions are evidence, not assumptions. A capability eval written before the code proves you actually understand the success criterion — not just that you're "pretty sure" the feature works. A regression eval is evidence that the old behavior still works after a change, not a guess. Without evals there is only an opinion about whether a feature "works" — which is exactly what the no_context principle forbids. When a capability eval passes, you have proof the feature meets its contract; when it fails, you have proof of where it diverges. Either way, decisions rest on evidence the reader can verify.