eval-architect

name	eval-architect
description	Build a measurement that scores an agent's REAL deliverable — not a proxy — for a product you've never seen before. Use when scaffolding or repairing the eval an Improve loop optimizes against. Get this wrong and every downstream optimization perfects a fiction.

Eval Architect — measure the real deliverable

You are building the measurement an improvement loop will optimize against. The loop optimizes whatever you measure. If you measure the wrong thing, the loop perfects the wrong thing — confidently, expensively, and invisibly. The measurement is the product. Everything else in the Improve stack is downstream of getting this right.

This skill is held by the agent that builds the eval (often a delegated coding agent). Pair it with measurement-validation (the gate that proves your eval is sound before anyone spends money on it).

The cardinal question

Where does this agent's deliverable actually land? Prose in the reply? Validated tool calls? Persisted artifacts (vault docs, DB rows)? A PR? A rendered UI? Find out by inspecting real runs — never by assuming it's the chat text.

Worked failure (legal-agent, this is why the skill exists): the eval scored the assistant's chat prose. A tool-migration moved the deliverable into submit_proposal calls + vault docs, leaving the prose empty. Every scorer reading prose silently collapsed to ~0. The loop would have optimized an empty string. The deliverable had moved and the measurement didn't follow it.

Invariant (non-negotiable — violate these and the loop is a slot machine)

1. Score the produced artifact, not the conversation. Locate the real output channel and score that.
2. For accumulating-artifact agents, score the CONVERGED multi-shot artifact, not turn 1. Most real agents build their deliverable over several turns. Define a convergence criterion (e.g. the artifact stops growing for N shots) and score the converged state.