prompt-eval-and-regression

name

prompt-eval-and-regression

description

Build the eval/regression set that gates prompt changes — labeled input/expected pairs over the hard cases, a scoring method (exact / schema-valid / rubric / LLM-judge with its caveat), a pass threshold, a CI gate with the model pinned, and injection cases. Reach for this before shipping a prompt, when a tweak silently broke other cases, or to defend against prompt injection. Pairs with structured-output-design.

Skill: Prompt eval & regression

Turn "it worked when I tried it" into "it is verified to still work." A prompt is unverified until the regression set is green.

Step 0 — One opinion up front

Prompts are code. They live in version control, change in reviewable diffs, and must pass an eval before merge — exactly like any other code.

Step 1 — Build the regression set

Collect labeled input → expected pairs covering:

• The known-hard/edge cases.
• Every past failure (each production bug becomes a permanent test).
• Injection/jailbreak cases (see §4 of the decision-trees doc). Keep it in the repo next to the prompt.

Step 2 — Choose a scoring method

Match the method to the task:

• Exact / schema-valid — closed-form or structured tasks. Cheapest, most reproducible; prefer it wherever possible.
• Rubric — explicit criteria, human or model graded.
• LLM-as-judge — scalable but the judge is a fallible prompt. Judge the judge: validate against human labels first; never let a model grade its own output unaudited; watch for position/verbosity/self-preference bias. Set a pass threshold a change must clear.

Step 3 — Pin for reproducibility

Pin the model + version + temperature (+ seed where available). Handle nondeterminism: temperature 0 where the task allows; multiple samples + a tolerance where it doesn't. An eval against an unpinned model proves nothing tomorrow.

Step 4 — Wire the CI gate

Run the regression set on every change to a prompt file; fail the build on a regression. State honestly what the set does and doesn't cover — a green gate over a thin set is false safety.

Step 5 — Version & roll out

Prompts carry a version (semver or content hash). Roll changes out shadow → canary → full, with a watched metric and a rollback trigger.

Step 6 — Hand off

• The large offline eval program / benchmark design → llm-evaluation-engineering.
• Adversarial attacks on the running system → ai-red-teaming.
• The fix for what the evals catch → prompt-implementation-engineer / prompt-architect.

Output

A regression set + scoring method + threshold, a CI gate with the model pinned, injection cases included, an honest coverage statement, and a versioning + canary rollout plan.