evaluating-agent-prompts

name	evaluating-agent-prompts
description	Evaluates agent prompts with human review, golden tests, and scoring rubrics, and is used when prompt quality must be measured and improved before demo use.

Creates a simple but disciplined prompt-evaluation loop so story quality improves instead of drifting.

Use this skill when

• Agent prompts are being written or tuned.
• The team needs a repeatable way to compare prompt versions.
• Human-in-the-loop review is required before shipping story behavior.
• The PM needs a simple review process that does not require deep coding knowledge.

Capabilities

• Defines golden test prompts for emotional intake and story generation.
• Creates rubrics for scoring prompt quality.
• Supports terminal-based or script-based evaluation runs.
• Encourages human review of the most important outputs.
• Tracks prompt revisions and why they changed.
• Focuses evaluation on what matters most for Diverge: story quality.

Workflow

1. Explain the purpose simply. Example: "We are testing prompts the way a product team tests messaging — by comparing outputs against a clear quality bar."
2. Collect the prompt files. Keep one prompt per agent or per major task.
3. Create golden test inputs. Include a balanced set of emotional cases, such as:
   • regret
   • grief
   • burnout
   • fear of disappointing others
   • conflict between honesty and love
4. Define rubrics. Score for:
   • emotional understanding
   • narrative depth
   • structural compliance
   • choice quality
   • clarity
   • originality
5. Run prompts consistently. Use the same inputs when comparing prompt versions.
6. Review outputs with a human. Especially for Emotional Cartographer, Story Architect, Choice Designer, and Continuity Editor.
7. Record what changed and why.
8. Promote only better prompts.

Suggested golden set

Start with 8–12 cases:

• short direct response
• vague response
• emotionally rich response
• contradictory response
• one upbeat but uncertain response
• one bittersweet response
• one difficult moral tension
• one low-signal input requiring follow-up

Example rubric

1–5 scale
- Understood the emotional core
- Produced meaningful, adult story material
- Stayed inside 5 anchors / 4 decisions / 4 endings
- Avoided trivial choices
- Felt original, not generic
- Was clear enough to hand off as typed JSON

Suggested lightweight workflow

• Store prompts in backend/agents/
• Store golden cases in docs/ or a small eval folder
• Review results in terminal or markdown tables
• Ask the PM for story-quality feedback at key revisions

Guardrails

• Do not tune prompts only for technical correctness.
• Do not skip human review just because JSON validates.
• Prompt evaluation is not optional in this project.

Project references

• AGENTS.md — prompts must be assessed, human-in-the-loop required, story quality is the product.
• docs/PRD.md — typed evaluation and narrative quality expectations.
• docs/roadmap.md — MVP success depends on one excellent path, so prompts must be polished.