design-eval-suite

name	design-eval-suite
description	Turn a fuzzy 'is the AI good enough?' into a concrete measurement plan: task spec, the eval method from the decision tree (exact / rule-based / model-graded / human), the metric + what it misses, the offline-vs-online split, a power-adequate sample size, and a numeric ship-gate decided before the run. Reach for it at the START of any LLM-feature eval. Used by `eval-strategy-lead` (primary).

Skill: design-eval-suite

Invoked by: eval-strategy-lead (primary). Hands off to eval-harness-engineer to build.

When to invoke: "how do I evaluate this?"; before writing any harness code; whenever an LLM feature has no measurement plan.

Output: a task spec + eval method + metric(s) + offline/online split + sample size + a numeric ship-gate.

Procedure

1. Write the task spec. Name the inputs, the desired output, and what "good" means to the user — not to a benchmark. If you can't state the failure a user would notice, you can't measure it.
2. Pick the method from the decision tree. Traverse ../../knowledge/eval-method-decision-tree.md: checkable answer → exact/programmatic; constrained format → rule-based (regex/schema/unit test); subjective quality → model-graded or human. Prefer the cheapest reliable method.
3. Choose the metric and name what it misses. Accuracy, F1, pass-rate, win-rate, groundedness — pick the one that tracks user value, then write one sentence on what it drops (so nobody Goodharts the proxy).
4. Split offline from online. Decide what the frozen golden set gates (ship/no-ship) and what only production can tell you (thumbs, escalation rate, downstream conversion). You need both.
5. Size the set. Enough examples that a real regression clears the noise — start ~100 real examples for a ship decision, more where the effect is small or the bar is high. Stratify across the input segments that matter.
6. Set the ship-gate as a number, before the run. "metric ≥ X on N examples, zero guardrail regressions." Decide X before seeing the score so it can't be rationalized after.

Worked example

Support-reply drafting feature.

• Task spec → input: ticket + KB; output: a draft reply; good = accurate, on-policy, no fabricated facts.
• Method → groundedness is subjective → model-graded judge; "did it cite a real KB article?" → rule-based check alongside.
• Metric → judge win-rate vs the current prompt + a hard groundedness pass-rate. Misses: tone nuance → sample to human monthly.
• Offline gate → judge win-rate ≥ 55% AND groundedness = 100% on 120 frozen tickets. Online → thumbs-down rate + agent-edit distance.

Guardrails

• Never gate on a leaderboard/benchmark score — it doesn't measure your task.
• Decide the threshold before the run — a bar set after seeing the number is not a gate.
• A vibe check is a hypothesis — fine to start, never to decide.