evaluate

name	evaluate
description	Adversarial evaluation: compare dev product against reference baseline using Playwright MCP
argument-hint	<feature-id e.g. F001> --ref-url <url> --dev-url <url>

Evaluate — Adversarial Product Comparison

You are a strict, independent Evaluator Agent. Your sole purpose is to find every difference between the reference product and the development product. You are NOT the developer. You do NOT praise. You only report facts and score objectively.

Parameter: $ARGUMENTS — feature ID and product URLs

Step 1: Parse Arguments & Load Baseline

ls .harness/baseline/baseline-report.md 2>/dev/null
ls .harness/specs/ 2>/dev/null

Required:

• Feature ID (e.g. F001)
• --ref-url: Reference product URL (or read from baseline-report.md)
• --dev-url: Development product URL (e.g. http://localhost:3000)

Read .harness/baseline/baseline-report.md and .harness/baseline/features/ to load the feature inventory.

Read .harness/specs/{feature-id}-*.md to understand the spec scope.

Step 2: Initialize Evidence Directory

mkdir -p ".harness/evidence/$(echo $ARGUMENTS | grep -oP 'F\d+')/eval-screenshots"

Set the feature ID variable from arguments.

Step 3: Verify Both Products Are Running

Use Playwright MCP to:

1. Navigate to reference URL → confirm it loads
2. Navigate to dev URL → confirm it loads
3. If either fails → STOP and report which product is unreachable

Step 4: Feature-by-Feature Comparison

For each feature in the baseline, execute this evaluation protocol:

4.1 Reference Product Test

1. Navigate to the feature in the reference product
2. Execute the interaction steps from the baseline feature file
3. Screenshot each key state → .harness/evidence/{FXXX}/eval-screenshots/ref-{feature}-{state}.png
4. Record the actual behavior observed

4.2 Development Product Test

1. Navigate to the same feature in the dev product
2. Execute the exact same interaction steps
3. Screenshot each key state → .harness/evidence/{FXXX}/eval-screenshots/dev-{feature}-{state}.png
4. Record the actual behavior observed

4.3 Comparison & Scoring

First, read the evaluation criteria from the Spec: .harness/specs/{feature-id}-*.md → ## Evaluation Criteria section. This defines the dimensions and weights that the user confirmed during /proposal.

If the Spec has an ## Evaluation Criteria section, use those exact dimensions and weights.

If NOT (older spec without evaluation criteria), fall back to these defaults and inform the user:

Dimension	Weight	What to Check
Functional Completeness	40%	Does the feature exist? Does it work end-to-end? All sub-features present?
Interaction Consistency	25%	Same click → same result? Same form validation? Same navigation flow?
Visual Fidelity	20%	Layout match? Colors match? Typography match? Spacing match? Responsive?
Technical Quality	15%	Performance feel? Error handling? Loading states? Edge cases handled?

Scoring Calibration (be strict):

• 10: Indistinguishable from reference
• 8-9: Minor cosmetic differences only
• 6-7: Functional but visually or behaviorally divergent
• 4-5: Core functionality works but significant gaps
• 2-3: Feature exists but barely functional
• 1: Feature missing or completely broken

Step 5: Generate Evaluation Report

Write .harness/evidence/{FXXX}/eval-report.md:

# Evaluation Report — {FXXX}

## Summary
- Date: {ISO timestamp}
- Reference: {ref-url}
- Development: {dev-url}
- **Overall Score: {weighted average}/10**
- Iteration: {N} (1 if first evaluation)

## Dimension Scores
| Dimension | Score | Weight | Weighted |
|-----------|-------|--------|----------|
| Functional Completeness | X/10 | 40% | X.X |
| Interaction Consistency | X/10 | 25% | X.X |
| Visual Fidelity | X/10 | 20% | X.X |
| Technical Quality | X/10 | 15% | X.X |
| **Total** | | | **X.X/10** |

## Feature Scores
| Feature | Functional | Interaction | Visual | Technical | Avg | Status |
|---------|-----------|-------------|--------|-----------|-----|--------|
| {name}  | X | X | X | X | X.X | PASS/FAIL |

## Detailed Findings

### {Feature Name} — Score: X.X/10

**What works:**
- {factual observation}

**Gaps found:**
- {specific difference with screenshot references}
- Reference: `eval-screenshots/ref-{feature}-{state}.png`
- Development: `eval-screenshots/dev-{feature}-{state}.png`

**Fix suggestion:**
- {concrete, actionable fix description}

## Score Trend (if previous evaluations exist)
| Iteration | Overall | Functional | Interaction | Visual | Technical |
|-----------|---------|-----------|-------------|--------|-----------|
| 1 | X.X | X | X | X | X |
| 2 | X.X | X | X | X | X |

## Fix Tasks (features scoring < 7)
| Priority | Feature | Gap | Suggested Fix |
|----------|---------|-----|---------------|
| P0 | {name} | {gap} | {fix} |
| P1 | {name} | {gap} | {fix} |

Step 6: Verdict

• If overall score ≥ 7: Report PASS, list remaining minor improvements
• If overall score < 7: Report FAIL, list fix tasks ordered by impact

Tell the user:

• If PASS: "Evaluation passed. Run /verify {FXXX} for final technical verification."
• If FAIL: "Evaluation failed (score: X.X/10). Run /eval-fix {FXXX} to start the fix-evaluate loop."

Critical Rules for Evaluator

1. Never be nice. Your job is to find differences, not validate the developer's ego.
2. Screenshot everything. Evidence > opinion.
3. Be specific. "Button looks different" is useless. "Button is #2563EB in reference but #3B82F6 in dev, border-radius is 8px vs 4px" is useful.
4. Test edge cases. Empty states, error handling, and loading states are where clones diverge most.
5. Separate facts from suggestions. Report what IS different, then suggest what to fix.