eval

name

eval

description

Run the context fitness-function probe suite (.oh/evals/probes/*.sh) against real state and write the .oh/evals/RESULTS.md benchmark. Each probe is a deterministic 3-state oracle (PASS/REGRESSION/SKIPPED); a green→red transition is surfaced as a REGRESSION naming the lesson it closes. Tier-B behavioral evals are out of scope. TRIGGER when: asked to run evals, check the probe suite, "run /eval", verify a lesson's probe is green, benchmark the harness, or before/after editing a rule/skill that a probe guards.

Eval

The runner for the harness fitness function. It discovers .oh/evals/probes/*.sh, runs each against real state, and writes the .oh/evals/RESULTS.md scoreboard. A rectification is provably "done" when its probe is green; a recurrence shows up as a REGRESSION (was-PASS, now-fail) naming the # source: lesson. The full contract — 3-state exit oracle, header convention, correction-surface triage — is in .oh/evals/README.md.

Usage

bash .claude/skills/eval/run.sh                 # run the whole suite, rewrite RESULTS.md
bash .claude/skills/eval/run.sh --probe <id>    # run one probe, update only its row
bash .claude/skills/eval/run.sh --tier A        # run only Tier-A probes

Exit-code oracle (per probe): 0=PASS, 1=REGRESSION, 2=SKIPPED (not applicable — excluded from pass-rate), 124=TIMEOUT, other=ERROR. Each probe is wrapped in timeout 30s. Runner aggregate exit (the process $? of run.sh itself): 0 when no new green→red regression occurred this run, 1 when one or more new regressions were detected (${#regressions[@]} > 0). When invoked via the Bash tool as bash .claude/skills/eval/run.sh, the agent caller reads $? directly to gate on success — the printed REGRESSIONS (...) stdout block and per-probe stderr lines remain the human-readable signal. Note: the eval-weekly cron is an intentional legacy caller that appends || true then greps stdout; it does not consume the exit code by design — this is not a bug.

What the runner does

1. Recover orphaned ablation backups (M-2): if .oh/evals/.ablation-active exists from a crashed ablation, restore each <target>.bak before running.
2. Discover + run every probe matching the filters; extract # tier: / # source: via the exact header grep.
3. Compute the delta vs the prior RESULTS.md row. First run (no prior row) emits new-pass/new-fail and raises NO regression without prior state.
4. Surface regressions — any PASS → (REGRESSION|TIMEOUT|ERROR) transition is printed first, naming the probe's source.
5. Rewrite RESULTS.md atomically — build the full scoreboard into a temp sibling file (RESULTS.md.tmp.$$) and replace the live file in one mv -f (never truncate-then-append in place), so a crash or concurrent run can't leave a partial scoreboard. Overwrite the row for each probe run; carry prior rows for probes not run this invocation from a pre-write snapshot (RESULTS_ORIG) captured before the rewrite — not the live file — so a filtered run never erases untouched rows and the scoreboard stays complete.

Ablation (added by US-006)

run.sh --ablate <context-file> --probe <id> runs a probe with and without a target rule/memory file loaded (reusing scripts/ablate.sh's swap/restore/trap mechanics — NOT the claude -p oracle) and reports LOAD-BEARING (regression on removal) or PRUNABLE. This is the prune-half of the fitness function. See US-006 in .oh/tasks/context-fitness-evals/prd.md.

When NOT to use

• Tier-B behavioral evals (sub-agent + LLM-judge of judgment-call behavior) are deferred — /eval is deterministic only. Never hard-gate on a noisy metric.
• For scoring context files for staleness/budget, that is /context-audit and /skill-lint — /eval checks behavior/state, not prose quality.

Memory Protocol

After a run, append to .oh/memory/<UTC-date>/log.md per .oh/skills/retro/references/memory-protocol.md:

## eval -- HH:MM UTC
- **Result**: OP
- **Ran**: <n> probes (<p> PASS / <r> REGRESSION / <s> SKIPPED)
- **Regressions**: <named, or none>
- **Observation**: <one sentence>