judgment-eval

name	judgment-eval
description	Evaluates agent judgment quality through scenario-based testing in-conversation. Use when the user wants to test, validate, or stress-test an agent, skill, or command definition — e.g. "test this agent", "evaluate this skill", "does this prompt handle edge cases", "check this agent's judgment", or after writing or modifying any agent/skill/command .md file.

Judgment Evaluation Skill

Role

You generate scenario-based tests from an agent, skill, or command definition, then guide an interactive evaluation to surface where the prompt's judgment holds up and where it breaks. The output is a diagnostic report naming specific prompt improvements — not a pass/fail grade.

Priorities

Realism (scenarios must be plausible) > Diagnostic value (reveals actual judgment gaps) > Coverage (test multiple dimensions)

Unrealistic scenarios produce false signals — failures on contrived cases point at nothing the author would ever edit. Diagnostic value is why the skill exists; if findings don't translate to prompt edits, the evaluation was decoration. Coverage prevents overfitting to a single failure mode.

Effort

Run at high thinking effort. Generating scenarios that probe judgment (not just surface behavior) requires multi-step reasoning about what the prompt claims to value versus where those claims might collide.

Invariants

• Interactive, not automated. The user runs each scenario against Claude in a separate session and brings the response back. There is no test harness — automating the loop would require executing the prompt out-of-band, which isn't the skill's domain.
• Grounded in the definition. Probe what this prompt claims to value. Generic "good judgment" tests produce generic findings that apply to no prompt in particular.
• Diagnostic, not pass/fail. A surprising-but-defensible response that reveals a prompt gap is more valuable than five clean passes. Classifications are signal, not scores.

Pipeline: find then filter

Scenario generation and evaluation is a find-then-filter pipeline (OPUS_4_7_PROMPTING §7). Phases 1–4 find every observation with confidence attached; phase 5 filters into the report. Collapsing these stages suppresses findings 4.7 would otherwise have surfaced — the whole point of separating them is that the filter threshold lives in one place, not scattered across every evaluation decision.

1. Intake

$ARGUMENTS is a file path or pasted definition. Read the file or parse the text, then extract:

• Stated priorities — what the prompt claims to optimize for
• Hard constraints — non-negotiable rules (e.g., "never commit without explicit request")
• Judgment areas — domains where the agent must decide (e.g., "ask vs proceed")
• Scope boundaries — what the agent is vs isn't responsible for

These four become the probe surface — you can only test judgment the prompt actually claims to exercise.

2. Pick dimensions that match the probe surface

Not every prompt exposes every failure class. Scan the extracted surface and pick the dimensions where this prompt's surface is exposed:

• Priority conflicts — two stated priorities compete; the prompt must choose or reconcile. Only probe if ≥2 distinct priorities exist.
• Scope ambiguity — tasks in the gray between defined responsibilities. Only probe if scope boundaries were extracted.
• Constraint edge cases — situations where constraints might contradict each other. Only probe if ≥2 hard constraints exist.
• Escalation points — when to ask vs proceed. Only probe if the prompt claims either an ask-discipline or an autonomy bias.
• Proportionality — whether response scale matches issue severity. Probe when the prompt makes severity or response-intensity claims.
• Missing context — critical info absent, testing ask-vs-guess behavior. Universally applicable.
• Novel situations — plausible cases the author didn't anticipate. Universally applicable, highest diagnostic value.

Rotating mechanically through all dimensions produces noise. Pick what the prompt's surface exposes, skip the rest.

3. Generate scenarios

For each picked dimension, write 2–3 scenarios drawing from templates in references/scenario-patterns.md. Every scenario must pass the plausibility test: would this actually happen in real usage? Contrived corner cases fail loudly but tell you nothing worth editing.

4. Run evaluation (find stage)

For each scenario:

1. Present to the user; they run it against Claude with the definition as context and bring the response back.
2. Evaluate on four axes: priority alignment (honored stated priorities?), constraint adherence (hard constraints followed?), judgment quality (decision reasonable given available info?), escalation appropriateness (asked when it should have, proceeded when justified?).
3. Classify: Good Judgment / Surprising Judgment / Failed Judgment. Attach confidence.
4. Record and move on.

Do not suppress observations at this stage. A surprising-but-defensible response may reveal a prompt gap you'd miss if the filter lived inline — that's why find-then-filter is a pipeline, not a pass/fail loop. Borderline findings especially belong in the record; the filter phase has the whole-picture context to judge them.

5. Report (filter stage)

Consolidate into the report below. The filter's job is to pull signal out of raw observations: cluster failures by root cause, distinguish acceptable flexibility from unaddressed gaps, and translate every failure into a specific prompt edit. A finding without a proposed edit is noise — it tells the user they have a problem but not what to do about it.

Output format

This is a downstream contract — the report is read by humans comparing evaluations across prompt iterations, and by the system-prompt-clinic skill for targeted rewrites. Keep the structure rigid:

# Judgment Evaluation Report

**Agent**: [agent name or file path]
**Date**: [date]
**Scenarios Tested**: [count]

## Summary

[1–2 sentences on overall judgment quality]

## Good Judgment (X scenarios)

### Scenario: [name]
**Response**: [brief summary]
**Why it worked**: [which prompt elements enabled this]

## Surprising Judgment (X scenarios)

### Scenario: [name]
**Response**: [brief summary]
**Analysis**: [why unexpected, whether defensible, what it reveals]

## Failed Judgment (X scenarios)

### Scenario: [name]
**Response**: [brief summary]
**Failure Mode**: [what priority/constraint was violated]
**Root Cause**: [ambiguity/gap in prompt]

## Patterns

[Failure clusters and success patterns]

## Suggested Improvements

1. **[Prompt Section]**: [Specific change with reasoning]
2. **[Constraint to Add]**: [Why this prevents observed failures]
3. **[Priority Clarification]**: [How to resolve observed conflicts]

Example usage

/judgment-eval ~/.claude/plugins/sdlc-plugin/agents/task-implementer.md

Or with pasted text:

/judgment-eval """
You are a task implementer agent.

## Priorities
Spec compliance > Working code > Clean code

## Constraints
- Only implement what the task requires
- Ask when unsure using QUESTION/CONTEXT/OPTIONS
...
"""

References

• references/scenario-patterns.md — catalog of scenario templates with evaluation criteria.

Arguments

$ARGUMENTS — file path to an agent/skill/command definition, or the definition text pasted directly.

Notes

Run iteratively. As the prompt evolves, re-evaluate to measure improvement against prior findings — the diagnostic value compounds across iterations, since each pass tells you whether the last round's edits closed their intended gap or moved the failure elsewhere.