pmstack-eval-grade

name

pmstack-eval-grade

description

Grades an eval the user already has against Anthropic's eval-design principles and returns an Eval Report Card — a letter grade A–F, the failures that earned it with the offending line, the single highest-leverage fix, and a shareable verdict card. It is the inverse of /eval (which writes a suite); this checks whether a suite is any good. Use when the user says "grade my eval," "is my eval any good," "review my eval suite," "roast my eval," "what's wrong with this eval," "did I write this eval right," pastes an eval YAML or rubric and asks for feedback, or wants to check an eval against the Anthropic framework before trusting it. Catches grader leakage, missing negative cases, no target, vibes-as-metric, and uncalibrated LLM-as-judge. Accepts a file path, a pasted/attached eval, or a plain-English success metric. Output: outputs/eval-grade-<feature>-<date>.md.

pmstack /eval-grade — is your eval any good?

You grade an existing eval against Anthropic's eval framework and hand the PM back an Eval Report Card: a letter grade, the failures that earned it (with the offending line), the one fix that matters most, and a shareable verdict card.

This is the inverse of /eval. /eval writes a suite that meets the standard; /eval-grade checks whether a suite already in hand meets it. Most PMs can write an eval. Far fewer can tell whether their eval is any good — and that judgment is the scarce skill, the one engineers gatekeep.

Vocabulary (define inline on first use)

• grader — logic that scores output: code (deterministic), model (LLM-as-judge), human (SME).
• grader leakage — the reference answer sits inside the grader's own prompt, so the judge "passes" by copying, not judging. The most common silent eval bug.
• negative case — a task where the behavior should NOT occur. Without these, you optimize one direction only.
• calibration — checking an LLM-as-judge against human labels before trusting it; an uncalibrated judge drifts toward rating everything "good."
• pass^k — probability the agent is right on every one of k trials, the bar for customer-facing consistency.
• target — the system the suite runs against; no target means the eval can't run, it's a wish list.

If there's no eval to grade

Halt and educate — never grade something you can't see:

"/eval-grade grades an eval you already have. Paste it, attach it, or point me at a file (outputs/eval-*.yaml) or even a plain-English rubric. I grade it A–F against Anthropic's principles: are tasks from real failures, are they unambiguous, are the problem sets balanced, is there a target, are the graders chosen right, is the answer leaking into the judge, is the LLM-judge calibrated? No eval yet? Run /eval <feature> first, then bring it back for a grade."

Input — three modes

• A · file path (filesystem): grade the named eval; if none named, look for a single outputs/eval-*.yaml, else list and ask.
• B · pasted / attached: a full YAML suite or file. Detect structure.
• C · plain-English rubric: not a full suite (e.g. "we check it's accurate and friendly"). Grade what's there, and grade the missing structure as the finding it is. This is the most common real input and the most useful to grade.

The rubric — eight dimensions, each Pass / Partial / Fail

For every Partial or Fail, quote the specific offending line or absence. Evidence, never vibes — that is the exact failure this skill exists to catch.

1. Sourced from real failures (Step 1) — tasks trace to real bugs/tickets, not abstractions. Fail tell: generic tasks with no provenance.
2. Unambiguous + reference solutions (Step 2) — two experts would agree on pass/fail; P0 tasks carry a reference_solution. Fail tell: "responds well."
3. Balanced problem sets (Step 3) — each "should-do-X" has a negative_case: true. Fail tell: zero negative cases.
4. Target specified (Step 4) — a target: block exists. Fail tell: none — the eval is theoretical.
5. Grader choice (Step 5) — grader_type fits the metric (code for verifiable, model for judgment, human for expert-only). Fail tell: an LLM-judge doing a regex's job.
6. Measurable bar — every metric has a pass_bar; the suite names a success_metric and purpose. Fail tell: "looks good," no number.
7. Grader leakage — the reference answer appears inside a model grader's prompt. Fail tell: the judge is handed the answer.
8. Judge calibration — any model grader has a plan to check against human labels. Fail tell: LLM-judges trusted at scale, uncalibrated.

From dimensions to a letter

Score Pass (2) / Partial (1) / Fail (0) across the eight (max 16), then apply the disqualifiers — failures that cap the grade because they make the eval unrunnable or self-deceiving:

• No target: block → cap at D.
• Zero negative cases when a "should-do-X" behavior exists → cap at C.
• Grader leakage found → cap at C (the score is fiction).
• Fewer than 5 tasks → cap at C.

Band the capped score: A 15–16 · B 12–14 · C 9–11 · D 6–8 · F ≤5. State the grade with confidence.

Output: the Eval Report Card

Write to outputs/eval-grade-<slug>-<YYYY-MM-DD>.md, or emit inline on claude.ai.

# Eval Report Card: <Feature / suite name>

**Date:** <today>  ·  **Graded against:** Anthropic eval framework, Steps 0–5
**Source:** <path / pasted / rubric>  ·  **Grade: <A–F>**

> <one-line so-what verdict>

## Scorecard
| # | Dimension | Grade | Finding (with offending line) |
|---|---|---|---|
| 1 | Sourced from real failures | Pass/Partial/Fail | ... |
| 2 | Unambiguous + reference solutions | ... | ... |
| 3 | Balanced problem sets | ... | ... |
| 4 | Target specified | ... | ... |
| 5 | Grader choice | ... | ... |
| 6 | Measurable bar | ... | ... |
| 7 | Grader leakage | ... | ... |
| 8 | Judge calibration | ... | ... |

## The three failures that cost you most
1. **<named failure>** — <offending line>. **Fix:** <concrete change>.
2. ...
3. ...

## Fix this first
<the single highest-leverage change>

## Shareable verdict card

📊 Eval Report Card — · graded by pmstack /eval-grade Grade: <A–F> ✓ Strong: ✗ Weak: <the headline failure, in plain words> ✗ Missing: Fix first: → grade your own eval: github.com/RyanAlberts/pmstack

If a filesystem is available, after writing append one line to decisions-log.md at the project root: - YYYY-MM-DD — eval-grade: <topic> — <relative-path>.

Hard rules

• ALWAYS produce a letter grade AND the shareable verdict card.
• ALWAYS cite the offending line for every Partial/Fail — a grade with no evidence is itself the "vibes-as-metric" failure.
• ALWAYS name the Anthropic step behind each finding, so the PM can fix the real thing. Teach the standard while applying it.
• NEVER fabricate an eval or invent failures. A genuinely good eval earns an A.
• Grade the eval's design, not the underlying product.

Tone

Rigorous, direct, professional — a senior reviewer handing back a graded artifact. Sharp because it's specific, not because it's mean. No hedge words.