prompt-eval

name

prompt-eval

description

Evaluate and benchmark AI agent prompts with scientific rigor. Use when (1) assessing whether a prompt achieves its intended behavior, (2) comparing prompt versions with controlled experiments, (3) building an evaluation suite for prompt quality, (4) auditing existing prompts for weaknesses. Triggers on phrases like evaluate prompt, prompt benchmark, prompt audit, test prompt quality, compare prompts, prompt experiment.

Prompt Eval

Evaluate prompts like a scientist, not a critic.

Core Problem

Prompt quality has two layers that must be measured separately:

Layer 1 (Static):  Does the prompt text look right?   → Structural audit
Layer 2 (Dynamic): Does the prompt actually work?      → Running experiments

A prompt can score 5/5 on structure but fail at runtime. A prompt can look ugly but work perfectly. Only Layer 2 is ground truth.

The Two-Layer Evaluation Framework

Layer 1: Structural Audit (fast, cheap, necessary but insufficient)

Read the prompt text and score it on relevant design patterns. See references/audit-rubric.md for the full rubric.

This layer answers: "Is this prompt well-structured?"

When to use: Before running experiments. Catches obvious problems fast.

Limitation: Cannot predict actual behavior. A structurally perfect prompt may still fail.

Layer 2: Controlled Experiments (slow, expensive, ground truth)

Run the prompt against real tasks, measure outcomes, compare against controls. See references/experiment-design.md for the full protocol.

This layer answers: "Does this prompt actually produce the desired behavior?"

When to use: After Layer 1 passes. The only way to know if a prompt works.

Requirement: Must include a control group (v0 without the pattern being tested).

Experiment Protocol

Step 1: Define the task

Write a concrete, unambiguous task with measurable success criteria.

id: task-XXX
name: "Task name"
description: "What the agent should do"
expected_behavior:
  - "Specific observable behavior 1"
  - "Specific observable behavior 2"
scorer: deterministic | llm_judge | transcript_analysis
pass_criteria: "X/Y must pass"

Step 2: Write the prompts

Write two versions:

• v1: With the pattern you're testing
• v0: Without it (the control)

The only difference between v1 and v0 should be the pattern being tested.

Step 3: Run experiments

Run each version N times (N ≥ 3 for initial testing, N ≥ 10 for publication).

v1 → run 1, run 2, run 3, ... → results
v0 → run 1, run 2, run 3, ... → results

Step 4: Score results

Use deterministic scorers when possible. Fall back to LLM judges for subjective dimensions.

Every score must include evidence: actual output, not just "passed."

Step 5: Analyze

Compare v1 vs v0. Three possible outcomes:

v1 wins: Pattern is effective — keep it, refine it
v0 wins: Pattern is harmful or irrelevant — investigate why
Tie:     Pattern doesn't matter for this task — try harder tasks

Key Rules

• Control group is mandatory. Without v0, you can't attribute success to the pattern.
• Same model, same temperature, same task. Only one variable changes: the prompt.
• Evidence over assertion. "Passed" means showing the output, not claiming it.
• Failures are more informative than successes. When v1 loses, you learn something. When v1 wins, you might just have easy tasks.
• Static scores ≠ runtime scores. Never report a structural audit score as if it were an experiment result.

What Makes a Good Eval Task

1. Unambiguous success criteria — Two people should agree on pass/fail
2. Measurable — Can be scored by code or clear rules, not just vibes
3. Discriminating — Should separate good prompts from bad ones (not 100% pass rate)
4. Realistic — Should reflect actual use cases, not toy examples
5. Has a control — You need to know what happens WITHOUT the pattern