eval-harness

name	eval-harness
description	Formal evaluation framework for Codex sessions implementing eval-driven development (EDD) principles
tools	Read, Write, Edit, Bash, Grep, Glob
version	0.1.0
source	fork
checksum	3ec82f88cb2646f6e7fe871c124846c3946c9861ff79ec7d8aad9e419c0a1fe7
updated_at	2026-02-11T01:29:16.000Z
intent	testing
layer	quality

Eval Harness Skill

Native Subagent Protocol (Codex)

Codex supports native subagents. Delegate with spawn_agent, coordinate with send_input, collect via wait_agent, and clean up with close_agent.

Execution preference:

1. Use native subagents first for independent workstreams (parallel when possible).
2. Merge results in main thread and run final verification.
3. Fallback only when delegation is blocked: use the [ANALYST]/[ARCHITECT]/[EXECUTOR]/[REVIEWER] structure in a single response.

Minimal orchestration pattern:

spawn_agent -> send_input (optional) -> wait_agent -> close_agent

A formal evaluation framework for Codex sessions, implementing eval-driven development (EDD) principles.

Philosophy

Eval-Driven Development treats evals as the "unit tests of AI development":

• Define expected behavior BEFORE implementation
• Run evals continuously during development
• Track regressions with each change
• Use pass@k metrics for reliability measurement

Eval Types

Capability Evals

Test if Codex can do something it couldn't before:

[CAPABILITY EVAL: feature-name]
Task: Description of what Codex should accomplish
Success Criteria:
  - [ ] Criterion 1
  - [ ] Criterion 2
  - [ ] Criterion 3
Expected Output: Description of expected result

Regression Evals

Ensure changes don't break existing functionality:

[REGRESSION EVAL: feature-name]
Baseline: SHA or checkpoint name
Tests:
  - existing-test-1: PASS/FAIL
  - existing-test-2: PASS/FAIL
  - existing-test-3: PASS/FAIL
Result: X/Y passed (previously Y/Y)

Grader Types

1. Code-Based Grader

Deterministic checks using code:

# Check if file contains expected pattern
grep -q "export function handleAuth" src/auth.ts && echo "PASS" ||  


npm  -- --testPathPattern= &&   ||  


npm run build &&   ||