evaluate

name	evaluate
description	Run the application and verify sprint contract criteria via API tests, Playwright interaction, and schema validation.
argument-hint	[group-id]
context	fork
agent	evaluator

Evaluate Skill

Verify that the implemented group meets all sprint contract criteria by running live checks against the application: API calls, Playwright browser interaction, and schema validation.

Ultracode tip: Leave ultracode off here (/effort high). Evaluation is deterministic three-layer verification against a fixed contract — there is nothing to fan out.

---

Usage

/evaluate C

Evaluates group C's sprint contract. The group ID matches a node in specs/stories/dependency-graph.md and a file at sprint-contracts/{group}.json.

---

Prerequisites

Before running /evaluate, verify:

• sprint-contracts/{group}.json exists and is valid JSON.
• project-manifest.json exists with api_base_url, ui_base_url, and health_check fields.
• Docker stack is expected to be running. If it is not, the health check in Step 4 will catch this and produce a FAIL.

---

Execution Steps

Step 1 — Load Evaluation Patterns

Read the stack-matched verification reference under .claude/skills/evaluate/references/ (see the References section below) for the rigor that applies to this project's stack, plus any custom assertion helpers.

Step 2 — Load Sprint Contract

Read sprint-contracts/{group}.json. The contract contains:

• api_checks: list of HTTP endpoint checks.
• playwright_checks: list of browser interaction sequences.
• design_checks: list of visual and component checks (evaluated in Full mode only).
• architecture_checks.files_must_exist: list of file paths that must be present on disk.
• features: list of feature IDs this group satisfies.

Step 3 — Load Project Manifest

Read project-manifest.json. Extract:

• api_base_url — base URL for all API checks (e.g., http://localhost:8000).
• ui_base_url — base URL for Playwright navigation (e.g., http://localhost:3000).
• health_check — path to the health endpoint (e.g., /health).

Step 4 — Verify Docker Stack

Run a health check to confirm the application is live:

curl --retry 5 --retry-delay 3 -sf {api_base_url}{health_check}

If the health check fails after 5 retries, immediately record a FAIL with failure_layer: "docker" and stop. Do not proceed to API or Playwright checks. A broken stack is not a partial pass.

---

Layer 1 — API Checks

For each entry in api_checks:

1. Execute the request via Bash:

   curl -s -w '\n%{http_code}' -X {method} {api_base_url}{path}
   

   Include -H headers and -d body as specified in the check entry.

2. Parse the response: the last line is the HTTP status code; everything before it is the response body.

3. Verify status code matches expect.status. A mismatch is a FAIL for this check.

4. Verify the response body contains every string listed in expect.body_contains. A missing string is a FAIL for this check.

5. If the check entry contains a schema_ref field, validate the response body against the schema:

   python3 -c "
   import json, jsonschema, sys
   body = json.loads(sys.stdin.read())
   schema = json.load(open('specs/design/api-contracts.schema.json'))
   ref = schema['{schema_ref}']
   jsonschema.validate(body, ref)
   print('schema valid')
   " <<< '{response_body}'
   

   A schema validation error is a FAIL for this check.

Record each check as PASS or FAIL with the actual vs. expected values.

Debugging API Failures

Before reporting an API check as FAILED, read the server logs:

docker compose logs backend --tail=50 2>&1

Include the relevant error from the logs in the failure report. This gives the generator the actual stack trace, not just "got 500 instead of 200."

---

Performance Checks

For each performance_checks entry in the contract:

# Measure response time
time_ms=$(curl -s -o /dev/null -w "%{time_total}" -X {method} {api_base_url}{endpoint} | awk '{printf "%.0f", $1 * 1000}')

If time_ms > max_response_time_ms, report as WARN (not BLOCK — performance is advisory unless critical).

---

Layer 2 — Playwright Checks

Selector and assertion rules are single-sourced in .claude/skills/evaluate/references/playwright-patterns.md (the canonical Playwright reference, shared with /test). The rules below are a summary — defer to that file if they differ.

For each entry in playwright_checks:

1. Use Playwright MCP tools to execute the interaction sequence:

   • browser_navigate — navigate to a URL.
   • browser_click — click an element (use getByRole, getByText, or getByLabel; never CSS selectors).
   • browser_fill_form — fill form fields.
   • browser_snapshot — capture the DOM snapshot for assertion.

2. Execute each step in the order specified. Do not reorder or skip steps.

3. Verify each assertion listed in the check entry:

   • Element visible: confirm the element appears in the snapshot.
   • Text matches: confirm the exact or partial text is present.
   • URL: confirm browser_navigate landed on the expected path.

4. Use expect().toBeVisible() for visibility assertions. Never use waitForTimeout() — if an element is not immediately visible, the check fails.

5. Record each check as PASS or FAIL with a description of what was asserted and what was found.

---

Layer 3 — Design Checks (Full Mode Only)

Skip this layer entirely in Lean mode (the design-critic runs once at group end instead).

In Full mode, delegate to the design-critic agent:

• Pass the list of design_checks entries from the sprint contract.
• Pass the ui_base_url.
• The design-critic returns PASS/FAIL per check with visual evidence (screenshots or snapshots).

Record the design-critic's verdicts as-is. Do not override them.

---

Architecture Checks

For each path listed in architecture_checks.files_must_exist:

• Verify the file exists on disk at the given path.
• If the file does not exist, record a FAIL with the missing path.

This check does not require Docker to be running.

---

Layer 4 — Security Gate

The validator is not security-complete without this layer. Run it in Full and Lean modes — every mode runs the security gate.

1. Spawn the security-reviewer agent against the group's changed files (run it concurrently with Layers 1–2 when possible — it does not need the app running).
2. The agent writes specs/reviews/security-verdict.json ({ pass, block_severities, summary, findings[] }). Read it.
3. Determine the blocking threshold: use the sprint contract's contract.security_checks.block_severities if present, else the default ["critical", "high"].
4. The security gate FAILs if any finding's severity is in the blocking set (equivalently, security-verdict.json#pass === false). Medium/low findings are WARN/INFO — record them, do not fail on them.
5. If security-verdict.json is missing, treat that as a FAIL with failure_layer: "security" and reason "security-reviewer did not produce a verdict" — a missing scan is not a pass.

This layer does not require Docker. It is independent of the app being reachable.

---

Update features.json

After all checks complete, update features.json for every feature ID listed in the sprint contract's features array:

• passes: true if all checks for that feature passed, false otherwise.
• last_evaluated: current timestamp in ISO 8601 format.
• failure_reason: null if passing; otherwise a human-readable description of the first failure (e.g., "GET /users/1 returned 404, expected 200").
• failure_layer: null if passing; otherwise one of "api", "playwright", "design", "unit_test", "docker", "security".

Do not remove existing fields from features.json. Merge the updates into the existing structure. Only update evaluation state fields: passes, last_evaluated, failure_reason, and failure_layer. Preserve immutable feature identity/specification fields such as id, category, story, group, description, and steps.

---

Write Evaluator Report

Write the full evaluation report to specs/reviews/evaluator-report.md:

# Evaluator Report — Group {group}

Date: {ISO 8601 timestamp}
VERDICT: PASS | FAIL

## API Checks

- [PASS] POST /users → 201 ✓
- [FAIL] GET /users/1 → expected 200, got 404
- [PASS] DELETE /users/1 → 204 ✓

## Playwright Checks

- [PASS] Upload page renders ✓
- [FAIL] Submit button not clickable
- [PASS] Success message visible after form submit ✓

## Design Checks

- [PASS] Button uses primary color token ✓
- [SKIP] Design checks skipped (Lean mode)

## Architecture Checks

- [PASS] All expected files exist ✓
- [FAIL] Missing: src/repository/user-repository.ts

## Security Gate

- [FAIL] VULN-001 (high): SQL injection in src/api/users.ts:47
- block: 1, warn: 2, info: 0 → gate FAIL

## Features Updated

- F001: PASS
- F002: FAIL (api: GET /users/1 expected 200, got 404)
- F003: PASS

The overall VERDICT is PASS only if every check across all layers passes and the security gate passes (security-verdict.json#pass === true). A single FAIL in any layer — including an open BLOCK (critical/high) security finding — produces a FAIL verdict.

---

Mode Behavior

Mode	Layer 1 (API)	Layer 2 (Playwright)	Layer 3 (Design)	Layer 4 (Security)
Full	Run	Run	Run	Run
Lean	Run	Run	Skip	Run

Determine the current execution mode from project-manifest.json field execution.default_mode (full/lean), or the --mode override when invoked under /auto or /build. Default to Full if absent. Note: this is distinct from verification.mode (docker/local/stub), which controls how the app is reached — do not confuse the two.

---

Gotchas

• Never skip a check: Every entry in the sprint contract must be evaluated. Skipping a check to make the verdict green is not acceptable.
• Never rationalize failures: If the API returns 404 and the contract expects 200, that is a FAIL — not a "known issue" or "works on my machine." Record it as a FAIL.
• Use getByRole, not CSS: Playwright checks must use semantic locators (getByRole, getByText, getByLabel). CSS selectors break with minor UI changes and are not permitted.
• Use expect().toBeVisible(), not waitForTimeout(): Arbitrary timeouts hide real failures. If an element does not appear immediately, the check fails.
• Docker won't start — that's a FAIL: If the stack is unhealthy, record failure_layer: "docker" and stop. Do not attempt workarounds or partial evaluations.
• Do not modify sprint contracts: The contract is a read-only input. If the contract appears wrong, report it; do not edit it to make checks pass.

---

References

The evaluation reference pack (read by the evaluator and design-critic agents) lives under .claude/skills/evaluate/references/:

File	Contents
references/contract-schema.json	Sprint contract JSON schema
references/scoring-rubric.md	Design scoring rubric (4 criteria, weights, exemplars)
references/scoring-examples.md	Calibration anchors (score 5, 7, 9) — read before scoring
references/playwright-patterns.md	Selector patterns and assertion patterns for Layer 2 (canonical Playwright reference, shared with /test)
references/verify-python.md	Deep Python verification rigor (pytest/mypy/ruff, traceback parsing, FastAPI/async) — load when the backend is Python
references/verify-react.md	Deep React/TS verification rigor (build/tsc/vitest, browser/console/hydration signals) — load when the frontend is React/TS

Evaluator Behavioral Rules

These rules are non-negotiable. Deviation invalidates the evaluation.

1. Execute every check. Do not skip a check because a related check passed.
2. Never rationalize a failure. If the check specifies status: 200 and you get 201, that is a FAIL.
3. Evidence over opinion. Every verdict must cite specific output: response body, screenshot path, line number.
4. No partial credit on binary checks. API and Playwright checks are pass/fail.
5. Design scores are evidence-based. Cite what you observed, not what you assumed.
6. Do not infer intent. If the contract says check X and X is absent, the check fails.
7. Run checks in order. Layer 1 before Layer 2 before Layer 3, then the Layer 4 security gate.
8. Document every check result, even passing ones.
9. Security is a gate, not advice. The overall verdict is FAIL if specs/reviews/security-verdict.json#pass is false (any critical/high finding). A functional pass with an open BLOCK vulnerability is still a FAIL. The security-guidance plugin is advisory and does not satisfy this gate — the security-reviewer agent does.