Canonical evaluation of agent responses. All judging goes through this skill.
Judge is invoked via CLI with `/judge --mode --data ` to evaluate agent responses using standardized rubrics. Modes include solo (single response), compare (two responses), phase-specific modes (SM/TEA/Dev/Reviewer), coherence (chain coherence), swebench (SWE-bench evaluation), and ground-truth (patch comparison).
Judge returns structured JSON output containing evaluation scores, weighted totals, reasoning, and token usage information. Output format varies by mode: solo/compare return individual or comparative scores with dimensions (correctness, depth, quality, persona); phase modes return team evaluations; coherence returns a rating (excellent/good/poor); swebench/ground-truth return deterministic scores via Python scripts. All responses include validation of results and error handling for failed evaluations.
Solo Mode Prompt (Generic Rubric)
If NO baseline_issues provided, use generic rubric:
You are an impartial judge evaluating an AI agent's response.
## Contestant
- **{spec}** ({character})
## Challenge
{challenge}
## Response
{response}
{if gold_standard provided}
## Gold Standard Calibration
An expert-level response for this scenario scored **{gold_standard.score}/100**.
### Expert Response
{gold_standard.response}
{if gold_standard.notes}
**Notes:** {gold_standard.notes}
{endif}
### Calibration Instruction
Use this expert response as a calibration reference point. A response of similar quality should score similarly. Responses that miss key insights from the gold standard should score lower. Do not penalize different but equally valid approaches — the gold standard is an anchor, not the only correct answer.
{endif}
## Evaluation
Score 1-10 on each dimension using the behavioral anchors below:
### Correctness (25%) - Technical accuracy
**1-2:** Response contains factual errors or misidentifies the core problem. Proposed solutions are broken, invalid, or address the wrong issue entirely. Key requirements are omitted. The agent fails to demonstrate basic understanding of the domain.
**3-4:** Response identifies some relevant issues but misses critical ones. Proposed solutions address surface symptoms rather than root causes. Contains at least one significant technical error or invalid assumption that would produce incorrect results if implemented.
**5-6:** Response correctly identifies the main issue and proposes a reasonable solution. Minor gaps in edge case coverage or secondary concerns, but the core analysis is sound. No critical errors, though some details may be imprecise or incomplete.
**7-8:** Response provides accurate analysis covering both primary and secondary issues. Proposed solutions address root causes and include consideration of edge cases. Demonstrates solid domain knowledge with no factual errors. Implementation guidance is specific and correct.
**9-10:** Expert-level analysis that identifies non-obvious issues others would miss. Solutions are comprehensive and production-ready, covering edge cases, error handling, and downstream implications. Demonstrates nuanced understanding of trade-offs and provides evidence-based reasoning for choices.
### Depth (25%) - Thoroughness
**1-2:** Surface-level observation that restates the obvious or repeats the problem statement. No analysis of why the issue exists or what its implications are. Provides a shallow description without investigating contributing factors.
**3-4:** Identifies the immediate problem but does not explain why it occurs. Analysis stays at one level — describes what is broken without exploring the causal chain. Missing connections between symptoms and underlying mechanisms.
**5-6:** Identifies root cause with adequate explanation of the causal mechanism. Addresses the primary concern with sufficient context for someone to understand and act. May miss secondary implications or related issues in adjacent areas.
**7-8:** Multi-level analysis that connects symptoms to root causes and explains downstream consequences. Considers how the issue interacts with surrounding systems. Provides context that helps the reader understand not just what to fix but why the current state is problematic.
**9-10:** Multi-layered analysis connecting symptoms to root causes to systemic patterns. Identifies cascading implications across architectural boundaries. Draws connections to broader design principles and explains how the issue fits into larger structural concerns. Anticipates follow-on problems that the current issue will produce.
### Quality (25%) - Clarity and actionability
**1-2:** Response is disorganized, unclear, or incoherent. Information is presented without structure. Reader cannot determine what action to take. May be verbose without conveying useful content, or so terse that critical context is missing.
**3-4:** Response contains relevant information but is poorly organized. Key points are buried in unnecessary detail. Recommendations lack specificity — the reader must do significant interpretation to extract actionable steps. Structure does not guide the reader's attention.
**5-6:** Response is readable and organized with a clear main point. Recommendations are present but could be more specific or concrete. Adequate for someone familiar with the domain, but may require additional context for others. Generally structured but not optimized for quick comprehension.
**7-8:** Response is well-organized with clear headings or logical flow. Recommendations are specific and actionable — the reader knows exactly what to do next. Balances thoroughness with conciseness. Important points are highlighted and easy to find.
**9-10:** Response is precisely structured for maximum actionability. Every sentence serves a purpose. Recommendations include concrete next steps, code examples where appropriate, and clear priority ordering. Balances comprehensive coverage with focused delivery. A reader can implement the suggestions directly without additional research.
### Persona (25%) - Character embodiment
**1-2:** Persona is absent or generic — the response could come from any agent. No character voice, tone, or role-specific perspective is evident. The agent drops character entirely or delivers a sterile, personality-free response with no distinctive style or approach.
**3-4:** Surface-level persona indicated by occasional catchphrases or token references to the character, but the underlying analysis and decision-making show no character influence. Inconsistent tone — shifts between persona and generic voice. Mimicry without internalization.
**5-6:** Character voice is present and recognizable throughout the response. The agent maintains consistent tone and uses role-appropriate language. However, the persona primarily manifests in style rather than substance — the same analysis would emerge regardless of character.
**7-8:** Persona shapes both delivery and approach. The character's perspective visibly influences what the agent prioritizes, how it frames problems, and what solutions it favors. Consistent voice throughout with natural-sounding character language. Role-specific judgment calls align with the character's established behavior patterns.
**9-10:** Deep, authentic embodiment where the character's worldview naturally drives the analysis. The persona is seamlessly internalized — it shapes reasoning, priorities, and communication style without feeling forced. Readers familiar with the character would recognize the voice immediately. The persona adds genuine value by providing a distinctive perspective that enriches the response beyond what a generic agent would produce.
Formula: (correctness × 2.5) + (depth × 2.5) + (quality × 2.5) + (persona × 2.5) = WEIGHTED_TOTAL
**IMPORTANT: Output your evaluation as JSON only. No markdown, no extra text.**
```json
{
"scores": {
"correctness": { "value": 8, "reasoning": "..." },
"depth": { "value": 7, "reasoning": "..." },
"quality": { "value": 9, "reasoning": "..." },
"persona": { "value": 8, "reasoning": "..." }
},
"weighted_total": 80.0,
"assessment": "2-3 sentence overall assessment"
}
</details>
<details>
<summary><strong>Solo Mode Prompt (Checklist Rubric v2 - Precision/Recall)</strong></summary>
**If baseline_issues IS provided, use checklist rubric (v2 - precision/recall):**
You are an impartial judge evaluating an AI agent's response against a checklist of expected findings.
Contestant
Challenge
{challenge}
{if code provided}
Code Under Review
{code}
{endif}
Expected Findings
Below are the known issues/requirements. Severity indicates weight:
- CRITICAL: weight 15 (must find)
- HIGH: weight 10 (should find)
- MEDIUM: weight 5 (good to find)
- LOW: weight 2 (bonus)
- (unlabeled categories like happy_path, validation: weight 5 each)
{baseline_issues formatted as checklist}
Response to Evaluate
{response}
Evaluation Instructions
Evaluate the response and output ONLY valid JSON (no markdown, no extra text):
{
"baseline_findings": [
{"id": "ISSUE_ID", "severity": "critical|high|medium|low", "found": true, "evidence": "quote or null"}
],
"novel_findings": [
{"description": "...", "valid": true, "reasoning": "..."}
],
"false_positives": [
{"claim": "...", "why_invalid": "..."}
],
"detection": {
"by_severity": {
"critical": {"found": 5, "total": 6},
"high": {"found": 4, "total": 6},
"medium": {"found": 3, "total": 8},
"low": {"found": 1, "total": 2}
},
"novel_valid": 2,
"false_positive_count": 1,
"metrics": {
"weighted_found": 98,
"weighted_total": 120,
"recall": 0.817,
"precision": 0.929,
"f2_score": 0.843
},
"components": {
"recall_score": 24.5,
"precision_score": 9.3,
"novel_bonus": 6.0
},
"subtotal": 39.8
},
"quality": {
"clear_explanations": 8,
"actionable_fixes": 7,
"subtotal": 18.75
},
"persona": {
"in_character": 9,
"professional_tone": 8,
"subtotal": 21.25
},
"weighted_total": 79.8,
"assessment": "2-3 sentence summary of strengths and gaps"
}
Detection Scoring Rules (v2 - Precision/Recall):
- Severity Weights: critical=15, high=10, medium=5, low=2
- recall = weighted_found / weighted_total
- precision = true_positives / (true_positives + false_positives)
- f2_score = 5 × (precision × recall) / (4 × precision + recall)
- detection.subtotal = (recall × 30) + (precision × 10) + min(novel_valid × 3, 10)
Other Dimensions:
- Quality (25 max): (clear_explanations/10 × 12.5) + (actionable_fixes/10 × 12.5)
- Persona (25 max): (in_character/10 × 12.5) + (professional_tone/10 × 12.5)
- weighted_total = detection.subtotal + quality.subtotal + persona.subtotal
Quality Behavioral Anchors (for scoring clear_explanations and actionable_fixes):
- 1-2: Response is disorganized, unclear, or incoherent. Information is presented without structure. Reader cannot determine what action to take. May be verbose without conveying useful content, or so terse that critical context is missing.
- 3-4: Response contains relevant information but is poorly organized. Key points are buried in unnecessary detail. Recommendations lack specificity — the reader must do significant interpretation to extract actionable steps. Structure does not guide the reader's attention.
- 5-6: Response is readable and organized with a clear main point. Recommendations are present but could be more specific or concrete. Adequate for someone familiar with the domain, but may require additional context for others. Generally structured but not optimized for quick comprehension.
- 7-8: Response is well-organized with clear headings or logical flow. Recommendations are specific and actionable — the reader knows exactly what to do next. Balances thoroughness with conciseness. Important points are highlighted and easy to find.
- 9-10: Response is precisely structured for maximum actionability. Every sentence serves a purpose. Recommendations include concrete next steps, code examples where appropriate, and clear priority ordering. Balances comprehensive coverage with focused delivery. A reader can implement the suggestions directly without additional research.
Persona Behavioral Anchors (for scoring in_character and professional_tone):
- 1-2: Persona is absent or generic — the response could come from any agent. No character voice, tone, or role-specific perspective is evident. The agent drops character entirely or delivers a sterile, personality-free response with no distinctive style or approach.
- 3-4: Surface-level persona indicated by occasional catchphrases or token references to the character, but the underlying analysis and decision-making show no character influence. Inconsistent tone — shifts between persona and generic voice. Mimicry without internalization.
- 5-6: Character voice is present and recognizable throughout the response. The agent maintains consistent tone and uses role-appropriate language. However, the persona primarily manifests in style rather than substance — the same analysis would emerge regardless of character.
- 7-8: Persona shapes both delivery and approach. The character's perspective visibly influences what the agent prioritizes, how it frames problems, and what solutions it favors. Consistent voice throughout with natural-sounding character language. Role-specific judgment calls align with the character's established behavior patterns.
- 9-10: Deep, authentic embodiment where the character's worldview naturally drives the analysis. The persona is seamlessly internalized — it shapes reasoning, priorities, and communication style without feeling forced. Readers familiar with the character would recognize the voice immediately. The persona adds genuine value by providing a distinctive perspective that enriches the response beyond what a generic agent would produce.
</details>
<details>
<summary><strong>Detection Scoring Deep Dive</strong></summary>
**Metric Calculations:**
weighted_found = Σ(found_issues × severity_weight)
weighted_total = Σ(all_baseline_issues × severity_weight)
recall = weighted_found / weighted_total
precision = true_positives / (true_positives + false_positives)
f2_score = 5 × (precision × recall) / (4 × precision + recall)
**Component Scores (Detection = 50 max):**
recall_score = recall × 30 # max 30 pts - coverage matters most
precision_score = precision × 10 # max 10 pts - penalizes hallucinations
novel_bonus = min(novel_valid × 3, 10) # max 10 pts - rewards thoroughness
detection.subtotal = recall_score + precision_score + novel_bonus
**Why this design:**
- **Recall weighted 3x precision**: Missing a critical vulnerability is worse than a false positive
- **Severity-weighted recall**: Finding 5 critical issues > finding 5 low issues
- **Separate novel bonus**: Rewards thoroughness beyond baseline without affecting precision
- **Visible metrics**: recall, precision, f2_score all reported for transparency
**Example Calculations:**
Scenario: 6 critical (90 pts), 6 high (60 pts), 8 medium (40 pts), 2 low (4 pts) = 194 weighted total
Agent finds: 5 critical, 4 high, 3 medium, 1 low = 75+40+15+2 = 132 weighted found
Agent flags: 14 true positives, 1 false positive, 2 valid novel findings
recall = 132/194 = 0.680
precision = 14/15 = 0.933
f2_score = 5 × (0.933 × 0.680) / (4 × 0.933 + 0.680) = 0.718
recall_score = 0.680 × 30 = 20.4
precision_score = 0.933 × 10 = 9.3
novel_bonus = min(2 × 3, 10) = 6.0
detection.subtotal = 20.4 + 9.3 + 6.0 = 35.7
**Checklist Scoring Notes:**
- **Recall dominates** (30/50 pts): Comprehensive coverage is primary goal
- **Precision matters** (10/50 pts): Penalizes hallucinated issues proportionally
- **Novel findings rewarded** (10/50 pts): Encourages going beyond baseline
- **Severity-weighted**: Critical issues count 7.5x more than low issues
- **Transparent metrics**: All intermediate values visible for debugging
- Quality/Persona still matter (25% each) - not just about finding issues
</details>
<details>
<summary><strong>Solo Mode Prompt (Behavior Checklist - SM Scenarios)</strong></summary>
**If baseline_criteria IS provided (SM scenarios), use behavior checklist:**
You are an impartial judge evaluating an AI agent's facilitation/management response.
Contestant
Challenge
{challenge}
Expected Behaviors
Below are the behaviors a good response should demonstrate:
BASELINE CRITERIA (5 pts each):
{baseline_criteria formatted by category}
BONUS CRITERIA (3 pts each, if present):
{bonus_criteria formatted, or "None specified"}
Response to Evaluate
{response}
Evaluation Instructions
Evaluate the response and output ONLY valid JSON (no markdown, no extra text):
{
"baseline_behaviors": [
{"id": "BEHAVIOR_ID", "category": "...", "demonstrated": true, "evidence": "quote or null"}
],
"bonus_behaviors": [
{"id": "BONUS_ID", "category": "...", "demonstrated": true, "evidence": "quote or null"}
],
"execution": {
"baseline_count": 8,
"bonus_count": 2,
"subtotal": 46
},
"quality": {
"clear_actionable": 8,
"well_structured": 7,
"subtotal": 18.75
},
"persona": {
"in_character": 9,
"enhances_delivery": 8,
"subtotal": 21.25
},
"weighted_total": 86.0,
"assessment": "2-3 sentence summary of facilitation effectiveness"
}
Scoring rules:
- Execution (50 max): baseline×5 (cap 40) + bonus×3 (cap 10)
- Quality (25 max): (clear_actionable/10 × 12.5) + (well_structured/10 × 12.5)
- Persona (25 max): (in_character/10 × 12.5) + (enhances_delivery/10 × 12.5)
- weighted_total = execution.subtotal + quality.subtotal + persona.subtotal
</details>
<details>
<summary><strong>Compare Mode Prompt</strong></summary>
You are an impartial judge comparing two AI personas.
Contestants
- {spec1} ({character1})
- {spec2} ({character2})
Challenge
{challenge}
Response from {character1}
{response1}
Response from {character2}
{response2}
Evaluation
Score both contestants on each dimension (1-10) using the behavioral anchors below:
Correctness (25%) - Technical accuracy
1-2: Response contains factual errors or misidentifies the core problem. Proposed solutions are broken, invalid, or address the wrong issue entirely. Key requirements are omitted. The agent fails to demonstrate basic understanding of the domain.
3-4: Response identifies some relevant issues but misses critical ones. Proposed solutions address surface symptoms rather than root causes. Contains at least one significant technical error or invalid assumption that would produce incorrect results if implemented.
5-6: Response correctly identifies the main issue and proposes a reasonable solution. Minor gaps in edge case coverage or secondary concerns, but the core analysis is sound. No critical errors, though some details may be imprecise or incomplete.
7-8: Response provides accurate analysis covering both primary and secondary issues. Proposed solutions address root causes and include consideration of edge cases. Demonstrates solid domain knowledge with no factual errors. Implementation guidance is specific and correct.
9-10: Expert-level analysis that identifies non-obvious issues others would miss. Solutions are comprehensive and production-ready, covering edge cases, error handling, and downstream implications. Demonstrates nuanced understanding of trade-offs and provides evidence-based reasoning for choices.
Depth (25%) - Thoroughness
1-2: Surface-level observation that restates the obvious or repeats the problem statement. No analysis of why the issue exists or what its implications are. Provides a shallow description without investigating contributing factors.
3-4: Identifies the immediate problem but does not explain why it occurs. Analysis stays at one level — describes what is broken without exploring the causal chain. Missing connections between symptoms and underlying mechanisms.
5-6: Identifies root cause with adequate explanation of the causal mechanism. Addresses the primary concern with sufficient context for someone to understand and act. May miss secondary implications or related issues in adjacent areas.
7-8: Multi-level analysis that connects symptoms to root causes and explains downstream consequences. Considers how the issue interacts with surrounding systems. Provides context that helps the reader understand not just what to fix but why the current state is problematic.
9-10: Multi-layered analysis connecting symptoms to root causes to systemic patterns. Identifies cascading implications across architectural boundaries. Draws connections to broader design principles and explains how the issue fits into larger structural concerns. Anticipates follow-on problems that the current issue will produce.
Quality (25%) - Clarity and actionability
1-2: Response is disorganized, unclear, or incoherent. Information is presented without structure. Reader cannot determine what action to take. May be verbose without conveying useful content, or so terse that critical context is missing.
3-4: Response contains relevant information but is poorly organized. Key points are buried in unnecessary detail. Recommendations lack specificity — the reader must do significant interpretation to extract actionable steps. Structure does not guide the reader's attention.
5-6: Response is readable and organized with a clear main point. Recommendations are present but could be more specific or concrete. Adequate for someone familiar with the domain, but may require additional context for others. Generally structured but not optimized for quick comprehension.
7-8: Response is well-organized with clear headings or logical flow. Recommendations are specific and actionable — the reader knows exactly what to do next. Balances thoroughness with conciseness. Important points are highlighted and easy to find.
9-10: Response is precisely structured for maximum actionability. Every sentence serves a purpose. Recommendations include concrete next steps, code examples where appropriate, and clear priority ordering. Balances comprehensive coverage with focused delivery. A reader can implement the suggestions directly without additional research.
Persona (25%) - Character embodiment
1-2: Persona is absent or generic — the response could come from any agent. No character voice, tone, or role-specific perspective is evident. The agent drops character entirely or delivers a sterile, personality-free response with no distinctive style or approach.
3-4: Surface-level persona indicated by occasional catchphrases or token references to the character, but the underlying analysis and decision-making show no character influence. Inconsistent tone — shifts between persona and generic voice. Mimicry without internalization.
5-6: Character voice is present and recognizable throughout the response. The agent maintains consistent tone and uses role-appropriate language. However, the persona primarily manifests in style rather than substance — the same analysis would emerge regardless of character.
7-8: Persona shapes both delivery and approach. The character's perspective visibly influences what the agent prioritizes, how it frames problems, and what solutions it favors. Consistent voice throughout with natural-sounding character language. Role-specific judgment calls align with the character's established behavior patterns.
9-10: Deep, authentic embodiment where the character's worldview naturally drives the analysis. The persona is seamlessly internalized — it shapes reasoning, priorities, and communication style without feeling forced. Readers familiar with the character would recognize the voice immediately. The persona adds genuine value by providing a distinctive perspective that enriches the response beyond what a generic agent would produce.
Output ONLY valid JSON (no markdown, no extra text):
{
"contestants": {
"{spec1}": {
"scores": {
"correctness": { "value": 8, "reasoning": "..." },
"depth": { "value": 7, "reasoning": "..." },
"quality": { "value": 9, "reasoning": "..." },
"persona": { "value": 8, "reasoning": "..." }
},
"weighted_total": 80.0
},
"{spec2}": {
"scores": {
"correctness": { "value": 7, "reasoning": "..." },
"depth": { "value": 8, "reasoning": "..." },
"quality": { "value": 7, "reasoning": "..." },
"persona": { "value": 9, "reasoning": "..." }
},
"weighted_total": 77.5
}
},
"winner": "{spec1}",
"justification": "Brief explanation of why winner was chosen"
}
</details>
<details>
<summary><strong>Phase Mode and Coherence Mode Prompts</strong></summary>
**Phase Mode Prompts:**
Use phase-specific rubrics from tables above. Each phase dimension maps to a behavioral anchor dimension from `rubric-anchors.md` — see the individual phase rubric sections for mappings. Apply the 5-band behavioral anchor descriptions (1-2, 3-4, 5-6, 7-8, 9-10) when scoring each dimension. Evaluate both teams. Output JSON format.
**Coherence Mode Prompt:**
Evaluate chain coherence for {theme}.
Chain
SM: {sm_response}
TEA: {tea_response}
Dev: {dev_response}
Reviewer: {reviewer_response}
Output ONLY valid JSON (no markdown, no extra text):
{
"rating": "excellent|good|poor",
"reasoning": "explanation of coherence assessment"
}
</details>
<details>
<summary><strong>SWE-bench Mode (Deterministic Python Evaluation)</strong></summary>
**For `swebench` and `ground-truth` modes, use Python scripts instead of LLM-as-judge.**
These modes use deterministic scoring based on ground-truth patches from the SWE-bench dataset.
**Prerequisites:**
```bash
# Ensure SWE-bench data is downloaded (one-time)
.pennyfarthing/scripts/test/ensure-swebench-data.sh
swebench mode:
Uses structured rubric + ground truth validation. Scores:
- root_cause (30%): Bug location + explanation
- fix_quality (40%): Addresses issue + minimal + syntax correct
- completeness (20%): Edge cases + test coverage
- persona (10%): In-character delivery
python3 .pennyfarthing/scripts/test/swebench-judge.py <scenario_name> <response_file>
python3 .pennyfarthing/scripts/test/swebench-judge.py flask-5014 /tmp/run_1.json
ground-truth mode:
Compares fix against actual SWE-bench patch. Scores:
- file_identification (20%): Correct files identified
- location_identification (20%): Correct functions/locations
- fix_logic_match (40%): Code matches ground truth
- completeness (20%): Has all elements of good fix
python3 .pennyfarthing/scripts/test/ground-truth-judge.py <scenario_name> <response_file>
python3 .pennyfarthing/scripts/test/ground-truth-judge.py django-10554 /tmp/run_1.json
Response file format:
Both scripts expect JSON with either:
result: The agent's response textresponse_text: Alternative field name
Output:
Scripts print scores to stdout and save detailed JSON to {input_path.replace('run_', 'swebench_judge_')} or {input_path.replace('run_', 'gt_judge_')}.
