| name | eval |
| user-invocable | true |
| tags | ["eval","measurement","quality","meta","standard"] |
| model | sonnet |
| model-preference | sonnet |
| model-preference-codex | gpt-5.4-mini |
| model-preference-cursor | claude-sonnet-4-6 |
| args-schema | [{"flag":"--session","description":"session_id to evaluate (default: last completed session via the resolution cascade)"},{"flag":"--no-write","description":"Evaluate + report without appending to the eval journal (.orchestrator/metrics/eval.jsonl)"},{"flag":"--verify","description":"Re-evaluate a stored run-id and diff per-dimension for scoring drift (exit 1 on drift)"}] |
| description | Use this skill to run an honest session-process evaluation (Standard v1, aiat-llm-eval/1.0) — score the last completed orchestrator session against the pre-registered rubric-v1 dimensions, run /eval, evaluate this session, produce an eval report, or re-verify a stored eval run for reproducibility. Deterministic-first with an optional advisory LLM judge; never produces a global score.
|
Platform Note: State files use the platform's native directory: .claude/ (Claude Code), .codex/ (Codex CLI), or .cursor/ (Cursor IDE). Shared metrics + the eval journal live in .orchestrator/metrics/. See skills/_shared/platform-tools.md.
Eval Skill — Session-Process Evaluation (aiat-llm-eval/1.0)
On-demand, honest measurement of ONE completed orchestrator session against the
pre-registered rubric-v1 check set. The deterministic engine
(scripts/eval-session.mjs → scripts/lib/eval/engine.mjs) reads only local
metrics files (sessions.jsonl + events.jsonl), scores the five deterministic
dimensions, appends a session-eval record to the journal, and optionally
renders an HTML report. An opt-in LLM judge overlays two advisory dimensions.
The standard this skill implements is docs/eval/aiat-llm-eval-v1.md;
the frozen, content-hashed check set is skills/eval/rubric-v1.md.
Posture Contract (load-bearing — read before executing)
- No global score, by construction. The record has no overall/total/mean
field, and this skill never derives one. Report per-dimension verdicts only.
- Never guess. Missing source data yields
cannot-determine (a first-class,
non-error verdict) with an honest reason — never a fabricated pass/fail.
Do NOT "fill in" a missing KPI or infer a gate result the events do not show.
- Deterministic before judge. The five deterministic dimensions are complete
on their own. The judge (Phase 3) is opt-in, ADVISORY, and
uncalibrated in
v1 — never blend a judge verdict into the deterministic tally.
- Journal is SSOT; the report is a derived view. The append-only
.orchestrator/metrics/eval.jsonl is authoritative. The HTML report is
rebuildable from any stored record and is never authoritative over the journal.
--verify is the reproducibility proof. Re-scoring stored source data
reproduces the stored dimensions byte-for-byte (exit 0) or reports drift
(exit 1). This proves the SCORING replays — NOT that the model is deterministic.- Self-evaluation is labelled as such. The orchestrator scoring its own
session is a self-evaluation, not an independent audit.
Phase 0: Bootstrap Gate
Read skills/_shared/bootstrap-gate.md and execute the gate check. If the gate is
CLOSED, invoke skills/bootstrap/SKILL.md and wait for completion before
proceeding. If the gate is OPEN, continue to Phase 1.
Do NOT proceed past Phase 0 if GATE_CLOSED. There is no bypass. Refer to
`skills/_shared/bootstrap-gate.md` for the full HARD-GATE constraints.
Phase 1: Config & Argument Loading
1.1 Read Session Config
Read and parse Session Config per skills/_shared/config-reading.md. Extract the
eval block (scripts/lib/config.mjs returns it as config.eval, parsed by
scripts/lib/config/eval.mjs):
enabled: boolean (default false)
mode: 'warn' | 'off' (default 'warn')
judge: 'off' | 'haiku' | 'sonnet' (default 'off')
report: 'html' | 'none' (default 'html')
handle: string | null (default null)
On-demand /eval runs regardless of eval.enabled. The enabled flag gates
the AUTOMATIC session-end eval phase only — it does NOT gate this command (same
posture as /reconcile vs reconcile.enabled). mode: off is honoured as a
kill-switch only for the automatic phase; on-demand invocation still runs. If
eval.judge is off, skip Phase 3 entirely.
Parser gotcha: the eval: key-line itself MUST NOT carry an inline comment
(strict /^eval:\s*$/); a trailing # comment on that exact line makes the
parser skip the whole block and silently apply ALL defaults. Sub-key lines
tolerate inline comments.
1.2 Parse Arguments
Inspect $ARGUMENTS:
--session <id> → pass through to --session.--no-write → evaluate without appending to the journal (dry-run).--verify <run-id> → verification mode: skip Phases 2–4, run the CLI
--verify path (see Phase 6), report MATCH/DRIFT, done.
1.3 Capture the Model Id (honest provenance)
The record's model.source records HOW the model id was captured, precisely
because self-report is unreliable:
- If
$ANTHROPIC_MODEL is set in the environment, the engine reads it
automatically with source: env — env wins over the flag (precedence
env > flag). Do not pass --model-id in that case; let the engine resolve it.
- Otherwise the coordinator passes its own self-reported model id:
--model-id <self-reported-model-id> --model-source self-report.
Phase 2: Deterministic Run
Run the deterministic engine via its CLI. Default target is the last completed
session (resolution cascade); --session overrides.
node scripts/eval-session.mjs [--session <id>] --json \
[--model-id <self-reported-id> --model-source self-report] \
[--no-write]
- Do NOT pass
--metrics-dir for a real run — the engine defaults to the
live .orchestrator/metrics, the session being evaluated.
- The CLI captures the eval
timestamp (the one sanctioned clock read) and hands
it to the engine as a parameter, so the scoring path stays clock-free and
--verify-reproducible.
- Exit codes:
0 success · 1 user error (session not found) · 2 system error.
On exit 1 (e.g. "no completed session found"), surface the message and stop —
do not retry with fabricated inputs.
Parse the emitted JSON record. It carries dimensions[] (5 deterministic
entries), kpis{}, provenance.rubric_sha256 (non-null once rubric-v1.md
exists), model, harness, and run_id. Unless --no-write was passed, the
record is already appended to .orchestrator/metrics/eval.jsonl by the CLI.
Contamination check: if the human-render/summary reports a peer-overlapped
window, note it — verification-evidence and gate-health will read
cannot-determine for that reason (attribution is unsafe), which is correct, not
a defect.
Phase 3: Judge Overlay (ONLY when eval.judge != off)
The judge runs coordinator-side — AskUserQuestion and the Agent tool are
not available inside a dispatched subagent, so the judge is dispatched from the
coordinator thread using the read-only agent session-orchestrator:eval-judge
(model = eval.judge). Reference the API; do not reimplement scoring here:
import { runEvalJudge, mergeJudgeDimensions } from '$PLUGIN_ROOT/scripts/lib/eval/judge.mjs';
import { appendEvalRecord } from '$PLUGIN_ROOT/scripts/lib/eval/sink.mjs';
const { status, dimensions } = await runEvalJudge({
dispatchAgent,
record,
model: EVAL_JUDGE_MODEL,
budget: JUDGE_BUDGET_TOKENS,
});
const merged = mergeJudgeDimensions(record, dimensions);
appendEvalRecord(merged, { path: '.orchestrator/metrics/eval.jsonl' });
- Every judge dimension arrives
advisory: true + calibration_status: "uncalibrated" (the schema firewall rejects any other shape). Keep them
visibly separated from the deterministic five in the summary.
- If
runEvalJudge returns a non-ok status (e.g. dispatch failed), keep the
deterministic record as-is and note the judge was unavailable — the
deterministic evaluation is complete without it.
- When
--no-write was passed in Phase 2, do NOT append the merged record either.
The judge merge re-writes the record with the SAME run_id/timestamp, so a
later --verify <run-id> re-scores the deterministic dimensions from source
and diffs them; judge dimensions are advisory and excluded from the drift diff.
Phase 4: Report (ONLY when eval.report == html)
Render the derived HTML view from the record:
import { writeEvalReport } from '$PLUGIN_ROOT/scripts/lib/eval/report.mjs';
const res = writeEvalReport(record, { generatedAt: new Date().toISOString() });
- Output path:
.orchestrator/eval/reports/<run_id>.html (gitignored — a derived
view, rebuildable from the journal).
writeEvalReport NEVER throws; on res.ok === false surface the WARN reason
and continue (the journal record is unaffected — the report is derived).- Name the report path in the chat so the operator can open it.
- When
eval.report == none, skip this phase.
Phase 5: Chat Summary
Emit a compact, honest per-dimension summary. Status lines only — no global score.
## /eval — <session_id> (self-evaluation, aiat-llm-eval/1.0 · rubric-v1 · n=1, no CI)
Deterministic:
verification-evidence PASS <one-line evidence>
plan-fidelity PASS completion_rate=1.0 (score)
gate-health PASS <one-line evidence>
process-safety PASS <one-line evidence + guard-emission disclosure>
efficiency-kpis N/A (reported: duration=…s waves=… agents=… tok_in=… tok_out=… carryover=…)
Judge (advisory, uncalibrated) [only when eval.judge != off]:
instruction-adherence <verdict> advisory
report-quality <verdict> advisory
cannot-determine: <k> of 5 deterministic dimensions (<reasons>)
Report: .orchestrator/eval/reports/<run_id>.html
Journal: .orchestrator/metrics/eval.jsonl (appended: <yes|--no-write>)
Re-verify: node scripts/eval-session.mjs --verify <run_id>
Always report the cannot-determine share explicitly — a high abstention count
is an honest signal about missing telemetry, not a failure to hide. Always print
the --verify command as the reproducibility handle.
Phase 6: Verification Mode (--verify <run-id>)
When Phase 1.2 detected --verify, run ONLY:
node scripts/eval-session.mjs --verify <run-id> --json
- Exit
0 + { match: true } → the stored record re-scores identically across
all deterministic dimensions. Report MATCH with the dimension count.
- Exit
1 + { match: false, diffs } → scoring drift. Report the per-dimension
diff (id.field: stored=… fresh=…). Drift means the source data or the engine
changed since the record was written — investigate, do not overwrite.
--verify reproduces the stored model + timestamp verbatim (no env override),
so a MATCH is a real reproducibility proof of the scoring, not of model output.
Cross-Platform (Codex CLI / Cursor / Pi) — FA4
The deterministic core is pure Node CLIs (scripts/eval-session.mjs) plus Node
library modules (report.mjs, sink.mjs) — they run identically on every
platform. Only the judge phase needs harness-specific tooling.
- Codex CLI / Cursor / Pi: the
Agent tool (and AskUserQuestion) are
unavailable, so Phase 3 (judge) is SKIPPED with a one-line note
("judge phase skipped: requires the Agent tool, unavailable on <platform>").
Phases 2, 4, 5, 6 run unchanged — they are Node-only. See
skills/_shared/platform-tools.md § Agent Dispatch Pattern.
harness.platform on the record is resolved from $SO_PLATFORM
(falls back to claude-code) inside the engine — no skill action needed.- The deterministic five dimensions + the HTML report +
--verify are fully
available on all platforms; the judge overlay is a Claude-Code-only enrichment
in v1.
Anti-Patterns
- DO NOT derive, print, or imply a global/overall/aggregate score — the record
forbids one by construction and so does every report.
- DO NOT guess or "fill in" missing data — a missing gate result, KPI, or
completion_rate is
cannot-determine/null, never a fabricated pass or 0.
- DO NOT present the judge's advisory verdict as a measurement — it is
uncalibrated in v1 and must stay visibly separated from the deterministic
tally.
- DO NOT treat the HTML report as authoritative — the journal is the SSOT; the
report is a rebuildable derived view.
- DO NOT skip
--verify when reproducibility is in question — it is the
executable proof, and its MATCH/DRIFT exit code is the source of truth.
- DO NOT pass
--metrics-dir for a real run — that points the engine at
fixture data instead of the live session metrics.
- DO NOT call
runReconcile-style writes from a subagent — the coordinator
owns every eval.jsonl append (PSA-007).
