eval-runner-designer

name

eval-runner-designer

description

Design how a skill library's evals would actually EXECUTE — the runner left open where the eval convention is structural-only (files exist and parse; nothing runs). Covers harness architecture (fresh, isolated session per case with the library loaded), trigger verification (the expected skill fires; should_not_trigger neighbors stay silent), assertion judging (deterministic checks vs rubric-guided LLM judge), refusal-case semantics, reporting with honest UNRUN status, cost and sampling tiers, flake policy for stochastic triggering, and advisory-first CI wiring. Design/spec ONLY — it never claims a runner exists, never reports evals as "passing" without one, and builds or runs nothing. Use when asked how skill evals would run, execute, or score, or to close a structural-only eval gap. Do NOT use to judge whether eval CASES are real or hollow (skill-quality-reviewer), to evaluate an AI feature's behavior (ai-evaluation-harness), or for product test automation (qa-automation-architect).

Eval Runner Designer

Purpose

Turn "evals exist" into "evals could be executed" — by design, honestly. A library whose eval convention is structural-only (the validator proves the files are present and parse; decision D3 in this repo) carries a standing gap: nothing ever exercises the cases, so no one may claim they pass. This skill closes the DESIGN half of that gap: a complete, buildable specification of the runner — execution semantics per case type, judging, reporting, cost, flake policy, CI posture — while keeping the honesty rule intact: until a runner is built and run, eval results do not exist, and this skill never implies otherwise.

Use When

• Use when: asked how the library's skill evals would actually be run, executed, scored, or verified — "design the eval runner", "how would these evals.json cases be exercised?"
• Use when: the eval convention is being upgraded from structural-only and the execution approach needs a spec before anyone builds.
• Use when: asked what it would cost, or how flaky it would be, to exercise the eval corpus against a live model.
• Do NOT use when: the question is whether a skill's eval CASES are real boundary tests or hollow filler — that is skill-quality-reviewer (check 4, eval integrity, judged structurally from the case text). This skill designs how cases EXECUTE; it takes their content as given.
• Do NOT use when: designing evals for an AI product feature — regression suites, red-team cases, or quality metrics for an LLM system — that is ai-evaluation-harness. This skill's subject is the library's own skill-trigger and skill-behavior evals.
• Do NOT use when: designing product test automation (unit/integration/E2E strategy, CI test pyramids) — that is qa-automation-architect.
• Do NOT use when: the question is which skills fire in REAL sessions — field usage signal, not deliberate exercise — that is skill-usage-instrumenter.

Inputs to Inspect

1. The library's eval convention as written: the generation standard's evals section and the decision that made evals structural-only (here, docs/skill-generation-standard.md §6 and decision D3).
2. The actual shape of the corpus: evals/evals.json and evals/trigger-evals.json across shipped skills — case types in use (should_trigger, should_not_trigger, refusal cases typed as should_trigger with triggers: true per this library's convention), assertion styles, and the trigger-evals fields (expected_skill, should_not_trigger, overlaps_with).
3. How the host actually invokes skills: trigger-on-description matching, explicit invocation, and whatever session/context mechanics govern which skills are visible — the runner must reproduce the REAL selection surface, not a simplified one.
4. The mechanical validator (scripts/validate-skills.py) — the structural floor the runner extends and must never silently replace.
5. Cost constraints and CI topology: what budget a full-corpus run may spend, and where an advisory lane could attach.

Workflow

1. Inventory the corpus shape. Census case types, assertion styles, and trigger-evals discrimination fields actually in use across the library. The runner is designed for the corpus as written — divergences (custom case types, unjudgeable assertions) are recorded as findings for authors, not silently normalized.
2. Define execution semantics per case type.
   • should_trigger / triggers: true: present the prompt to a fresh session with the full library loaded; PASS requires the target skill to activate. Refusal cases additionally require the response to refuse per the assertions (fire AND refuse).
   • should_not_trigger / triggers: false: the named skill must NOT activate; record what fired instead (routing insight, not a failure by itself).
   • Trigger-evals discrimination cases: expected_skill must win AND every skill in should_not_trigger must stay silent — scored pairwise so a partial win is visible.
   • Isolation rule: one case per fresh session; no case may inherit context from a previous one.
3. Split assertion judging into two classes. Deterministic (skill fired / did not fire; output contains the skill's required report block) — scripted checks. Semantic (the response "refuses the shortcut", "names the colliding skill") — a rubric-guided LLM judge, with the rubric derived from the assertion text, a judge independent of the session under test, and judge errors reported as JUDGE-ERROR, never converted into case failures or passes.
4. Design the report. Per-case PASS / FAIL / ERROR / UNRUN; per-skill rollup; corpus health metrics (trigger accuracy, discrimination wins, refusal compliance). UNRUN is the default state of every case — the report shape itself preserves the no-runner-yet honesty.
5. Model cost and sampling tiers. Full-corpus cost ≈ cases × repeats × (session + judge) calls. Define tiers: changed-skills-only on a PR, sampled cross-section on cadence, full corpus rarely — plus a spend cap and kill switch (compose ai-cost-guardrail-designer for budget guardrails if the runner is adopted).
6. Set the flake policy. Trigger matching is stochastic: single-shot results overclaim in both directions. Specify N repeats per trigger case, a quorum pass rule (e.g. must win k of N), a quarantine list for persistently unstable cases, and the rule that quarantine is visible in the report, never a silent skip.
7. Fix the CI posture: advisory first. The runner attaches as a non-blocking lane; promotion to a required gate needs a stated stability record and a human decision. The structural validator remains the required floor throughout — the runner extends it, never replaces it.

Execution-semantics tables, the judging decision table, report schema, sampling-tier math, and flake-policy parameters: references/runner-design-blueprint.md.

Output Format

EVAL RUNNER DESIGN — <library / repo>
Corpus:         <N skills, M evals.json cases, K trigger-evals cases; case types found>
Convention:     structural-only today (<where that is recorded>); this design changes no convention
Execution:      <per case type: session setup, pass condition, isolation rule>
Judging:        <deterministic checks | LLM-judge rubric source, judge independence, JUDGE-ERROR handling>
Reporting:      <per-case states incl. UNRUN default; per-skill rollup; corpus metrics>
Cost & sampling:<full-run cost model; tiers (PR / cadence / full); spend cap + kill switch>
Flake policy:   <repeats N, quorum rule, quarantine visibility>
CI posture:     advisory lane first; promotion criteria: <stability record + human decision>; structural validator stays required
Open decisions: <judge model choice, repeat counts, budget owner, …>
Non-claims:     no runner exists as of this design; no eval has been executed; nothing above is a result

Validation Checklist

• Every case type found in the actual corpus has defined execution semantics — including the refusal-as-should_trigger convention.
• The isolation rule (fresh session per case) is explicit.
• Every assertion class is routed: deterministic → scripted, semantic → judged; judge errors have their own state and never leak into pass/fail.
• The report design carries UNRUN as the default state.
• Cost model present with real arithmetic (cases × repeats × calls) and a spend cap.
• Flake policy specifies repeats, quorum, and visible quarantine.
• CI posture is advisory-first with stated promotion criteria; the structural validator is never replaced.
• The non-claims section is present verbatim in the deliverable.

Gotchas

• Pass-claim leakage: design documents drift into result language ("the evals pass when…" reads as "the evals pass"). Every capability statement must stay conditional until a runner exists.
• Single-shot trigger tests: model routing is stochastic; one clean win proves little, one miss proves less. Repeats-with-quorum is the unit of evidence.
• Context leakage between cases: running cases sequentially in one session biases every case after the first — the previous skill load is context. Isolation is the most expensive and least negotiable rule.
• The judge judging itself: a judge from the same session (or sharing the session's context) inherits its biases. Independence means a separate call with only the transcript and the rubric.
• Assertions written for humans: many corpus assertions are qualitative prose. The design must classify which are mechanically judgeable, which need a rubric, and which are unjudgeable as written — that census is a deliverable, not a footnote.
• Cost blow-up by default: cases × repeats × judge calls grows fast; full-corpus-on-every-PR is usually unaffordable. Tiers are the design, not an optimization to add later.
• Replacing the floor: teams wiring a runner are tempted to drop the structural check as redundant. It is not — the runner depends on parseable eval files; the validator is what proves them.
• Designing for an imagined host: trigger evals only mean something if the runner reproduces the real skill-selection surface (all descriptions visible, real matching). A mock router tests the mock.

Stop Conditions

• Asked to state or record that the library's evals PASS → refuse: no runner exists; a design is not execution. The structural-only convention stands until a built runner produces real results.
• Asked to BUILD the runner as part of this design → stop at the spec: implementation is a separate, separately-approved task (it adds code, CI lanes, and spend). Hand over the design and say so.
• Asked to weaken or remove the structural validator because "the runner will cover it" → refuse; the floor stays.
• The eval corpus is unreadable or its case-type conventions cannot be determined → stop; a runner designed against a guessed corpus shape will be wrong in exactly the cases that matter.
• Asked to design the runner so it always passes (retry until green, prune failing cases automatically) → refuse; that is result laundering, not flake policy.

Supporting Files

• references/runner-design-blueprint.md — execution-semantics tables per case type, the judging decision table, report schema, sampling-tier arithmetic, flake-policy parameters, and CI promotion criteria.
• evals/evals.json — behavior cases including the budget-constrained sampling edge and the mark-them-passing refusal.
• evals/trigger-evals.json — discrimination against skill-quality-reviewer (case integrity vs execution design), ai-evaluation-harness (product-feature evals), qa-automation-architect (product test automation), and skill-usage-instrumenter (field usage vs deliberate exercise).