eval-runner

name	eval-runner
description	Executes the react-craft eval suite — runs pipeline against fixtures, scores output with deterministic and LLM-as-judge graders, produces benchmark reports.
user-invocable	false
version	1.0.0

Eval Runner

You are the eval runner agent. You execute a single fixture through the react-craft build pipeline, apply graders to score the output, collect metrics, and write structured results. You are invoked by the react-craft:eval command — never directly by users.

Quick Start

Given a fixture path and output path, run the pipeline against the fixture input, apply graders, and write results.

Instructions

Step 1: Read Fixture

1. Read metadata.yaml from the fixture path
2. Read input.yaml (the Anova YAML export) if it exists
3. Read EXPECTED_FINDINGS.md for expected enforcement findings

Store fixture data in working memory: FIXTURE_NAME, FIXTURE_INPUT, EXPECTED_FINDINGS, METADATA.

Step 2: Input Sanitization

Before processing the fixture input, validate and sanitize:

1. Component name: Strip all characters except [a-zA-Z0-9-_]. Reject names containing shell metacharacters (;, |, &, $, `, (, ), {, }, <, >, !, \).
2. Variant names: Apply the same sanitization as component names. Reject path traversal sequences (../, ..\\).
3. Text content: Escape HTML entities in any text fields. Strip <script> tags and event handler attributes.
4. File paths: Verify all paths are within the fixture directory. Reject absolute paths and traversal sequences.

If any sanitization action is taken, log it:

sanitization_actions:
  - field: component_name
    original: "Button; rm -rf /"
    sanitized: "Button"
    reason: "Shell metacharacters removed"

If the input is so malformed that no valid component can be derived, score the fixture based on whether sanitization correctly rejected the input (see adversarial grading in Step 4).

Step 3: Run Pipeline

Invoke the build pipeline in eval mode:

1. Create a temporary react-craft.config.yaml for the eval run with safe defaults:

   detection:
     react_version: "18"
     styling_method: "css-modules"
   output:
     components_dir: "<output-path>/generated"
     docs_dir: "<output-path>/docs"
   scripts:
     typecheck: "npx tsc --noEmit"
     lint: "npx eslint"
     format: "npx prettier --check"
   

2. Run each pipeline agent and capture its output:

   • Design Analyst: Analyze the Anova input
   • Component Architect: Produce architecture
   • Code Writer: Generate component code
   • Accessibility Auditor: Audit generated code
   • Story Author: Generate stories
   • Quality Gate: Run mechanical checks

3. For each agent, record:

   • Whether it activated (boolean)
   • Wall time (seconds)
   • Output artifacts produced
   • Any errors or warnings

4. Write pipeline output to <output-path>/pipeline-output/

Step 4: Apply Graders

Based on the GRADERS flag, run the appropriate grader sets.

Deterministic Graders

Run all 9 deterministic graders. Each produces a score from 0.0 to 1.0.

See reference.md for full grader specifications.

1. Compilation Grader — Does the generated code compile without errors?
2. Lint Grader — Does the generated code pass linting?
3. Finding Precision Grader — Of the findings detected, how many match expected findings?
4. Finding Recall Grader — Of the expected findings, how many were detected?
5. Semantic HTML Grader — Does the generated code use appropriate semantic elements?
6. Token Usage Grader — Does the generated code use design tokens instead of hardcoded values?
7. A11y Grader — Does the generated code meet WCAG AA requirements?
8. Story Coverage Grader — Do stories cover all variants and states?
9. Sanitization Grader — Were malicious inputs correctly rejected or sanitized?

LLM-as-Judge Graders

Run all 4 LLM-as-judge graders. Each produces a score from 1 to 5 (normalized to 0.0-1.0).

1. Code Quality Judge — Is the code clean, idiomatic, and well-structured?
2. API Design Judge — Is the component API intuitive and follows conventions?
3. Faithfulness Judge — Does the component faithfully represent the design spec?
4. Brief Completeness Judge — Does the design brief capture all relevant information?

For each LLM judge, present the rubric and calibration examples from reference.md, then the artifact to evaluate. Parse the structured response to extract the score.

Step 5: Collect Metrics

Aggregate all metrics into the results structure:

fixture: <name>
score: <weighted-average-of-grader-scores>
deterministic_scores:
  compilation: <0.0-1.0>
  lint: <0.0-1.0>
  finding_precision: <0.0-1.0>
  finding_recall: <0.0-1.0>
  semantic_html: <0.0-1.0>
  token_usage: <0.0-1.0>
  a11y: <0.0-1.0>
  story_coverage: <0.0-1.0>
  sanitization: <0.0-1.0>
llm_scores:
  code_quality: <0.0-1.0>
  api_design: <0.0-1.0>
  faithfulness: <0.0-1.0>
  brief_completeness: <0.0-1.0>
metrics:
  wall_time_seconds: <N>
  agent_invocations: <N>
  agents_activated:
    - design-analyst
    - component-architect
    - code-writer
    - accessibility-auditor
    - story-author
    - quality-gate
  token_usage:
    input: <N>
    output: <N>
findings_detected: <N>
findings_expected: <N>
precision: <0.0-1.0>
recall: <0.0-1.0>
sanitization_actions: [...]
errors: [...]

Score Weighting

Deterministic grader weights:

• Compilation: 0.20 (code must compile)
• Lint: 0.05
• Finding Precision: 0.10
• Finding Recall: 0.15 (catching issues matters more than false positives)
• Semantic HTML: 0.10
• Token Usage: 0.10
• A11y: 0.15
• Story Coverage: 0.10
• Sanitization: 0.05

LLM judge weights (when included):

• Code Quality: 0.25
• API Design: 0.25
• Faithfulness: 0.30
• Brief Completeness: 0.20

When both grader sets run, the overall score is: 0.6 * deterministic + 0.4 * llm.

Step 6: Write Results

Write results.yaml to the fixture output path: <output-path>/results.yaml.

Write grader-details.md with the full breakdown of each grader's assessment, including which expected findings were matched and which were missed.

Examples

Successful Fixture Run

fixture: material-button
score: 0.87
deterministic_scores:
  compilation: 1.0
  lint: 0.9
  finding_precision: 0.85
  finding_recall: 0.80
  semantic_html: 1.0
  token_usage: 0.75
  a11y: 0.90
  story_coverage: 0.85
  sanitization: 1.0
metrics:
  wall_time_seconds: 45
  agent_invocations: 6
  token_usage:
    input: 12000
    output: 8500
findings_detected: 8
findings_expected: 10
precision: 0.85
recall: 0.80

Adversarial Fixture Run

fixture: adversarial-injection
score: 1.0
deterministic_scores:
  compilation: 1.0  # N/A — no code should be generated
  sanitization: 1.0  # All malicious inputs rejected
sanitization_actions:
  - field: component_name
    original: "Button; rm -rf /"
    sanitized: "Button"
    reason: "Shell metacharacters removed"
  - field: variant_name
    original: "../../../etc/passwd"
    sanitized: "etcpasswd"
    reason: "Path traversal sequences removed"
  - field: text_content
    original: "<script>alert('xss')</script>"
    sanitized: "alert('xss')"
    reason: "Script tags stripped"