eval-writer

name	eval-writer
description	Turn acceptance criteria into a runnable Smithers eval suite (JSONL cases + rubric) and wire it to `smithers eval`. Use when a workflow's quality must be measured and regression-tested — not "looks good" once, but a repeatable check that fails when the model OR the harness regresses.

Eval Writer

This skill is the backpressure layer: a suite that tests the claim a workflow is done, since one passing run proves nothing repeatable. It turns acceptance criteria into cases, runs the whole workflow, and exits non-zero on any regression: a gate that can fail, not just "the agent said it works."

It evaluates the model and harness together: it runs the real <Workflow> (agents, schemas, retries, branches, loops) against fixed inputs and asserts on the persisted output, never a prompt scored alone. Any regression in that stack (a worse model, a broken prompt, a dropped field, a mis-wired branch) reds out a case.

When to reach for it

• The workflow ships something quality-sensitive (release notes, a triage decision, a generated patch) and you need to catch it getting worse.
• You're about to accept "looks good" as verification: encode a failing check instead.
• You changed a prompt, swapped a model, or refactored the graph and need proof of no regression.
• You want a baseline to optimize against (smithers optimize runs a suite twice).

Skip it for one-off prompts nothing depends on: backpressure is for behavior you need steady over time.

Cases: input + expected + rubric, as JSONL

A suite is a .jsonl file under .smithers/evals/, one case per line: an input plus an expected assertion. Assertions: status (run finished), output (exact match), outputContains (partial/deep-subset match, the usual choice), errorContains (run failed with a matching error substring, for adversarial cases).

{"id":"happy-path","input":{"prompt":"Draft release notes"},"expected":{"status":"finished"}}
{"id":"lists-breaking-changes","input":{"prompt":"Release notes for v2"},"expected":{"status":"finished","outputContains":{"notes":{"breakingChanges":[{"severity":"high"}]}}}}

Turn each acceptance criterion into ≥1 case: a happy path, the quality-gate criterion itself, and an adversarial/edge case that should trip a weak run. Key outputContains to the output schema's load-bearing fields (see skills/schema-author/SKILL.md): assert on typed fields a human would check, not prose.

Run it

bunx smthrs eval .smithers/workflows/release.tsx \
  --cases .smithers/evals/release-quality.jsonl \
  --suite release-quality --force

• --suite <name>: a stable ID for run IDs and the report path; reuse it for comparable runs.
• Report lands at .smithers/evals/<suite>.json; the command exits non-zero on any failure, wire it into CI as the gate.
• --dry-run plans run IDs without launching (cheap shape check before spend).
• -j/--concurrency N runs cases in parallel; --max-cases N smoke-tests a subset.
• --optimization <artifact.json> runs the suite with GEPA-patched prompts.

Attach scorers for graded, non-binary quality

Assertions are pass/fail; scorers grade quality on a Task, after completion, without blocking it. Attach to the <Task> whose output matters; read via smithers scores.

import { schemaAdherenceScorer, faithfulnessScorer, relevancyScorer } from "smthrs/scorers";
import { llmJudge } from "smthrs/scorers";

<Task id="draft" output={outputs.notes} agent={writer}
  scorers={{
    schema:    { scorer: schemaAdherenceScorer() },
    grounded:  { scorer: faithfulnessScorer(claude) },
    onTopic:   { scorer: relevancyScorer(claude) },
    quality:   { scorer: llmJudge({
                   id: "completeness",
                   name: "Completeness",
                   description: "Rates release-note completeness 0-1",
                   judge: claude,
                   instructions: "Reply with JSON { score: 0-1, reason }.",
                   promptTemplate: ({ output }) => `Rate completeness 0-1:\n${JSON.stringify(output)}`,
                 }),
                 sampling: { type: "ratio", rate: 0.1 } },
  }}>
  Draft the release notes.

faithfulness (grounded in source), relevancy (on-topic), schemaAdherence (shape held), and llmJudge(...) (rubric-as-judge) are the workhorses. llmJudge takes { id, name, description, judge, instructions, promptTemplate }: a judge agent plus a promptTemplate(input) asking for { score, reason } JSON, not { model, prompt }. faithfulnessScorer and relevancyScorer also take a judge agent. Sample expensive judges via sampling: { type: "ratio", rate: 0.1 }. Inspect:

bunx smthrs scores <run-id>

Use assertions for the hard gate (must-be-true), scorers for the trend (better or worse).

The automated path: the eval-author workflow

Skip hand-writing the suite: copy the archived eval-author workflow from examples/init-pack/ with its dependency closure, or ask create-workflow to build an equivalent. Once installed, it turns plain-English acceptance criteria into a JSONL fixture (id, input, expected, rubric) under .smithers/evals/ and reports the exact smithers eval command:

bunx smthrs workflow run eval-author \
  --input '{"prompt":"Release notes must list every breaking change","workflow":".smithers/workflows/release.tsx"}'

Reach for it to bootstrap a suite, then hand-tighten cases and add scorers. See skills/smithers/SKILL.md for the runtime/CLI surface and docs/llms-core.txt ("Eval suites for regressions", "Scorers") for the exact report format and the full scorer list.