adev-eval

name	adev:eval
description	Run a graduated evaluation harness (0-100) scoring implementation quality across four layers: deterministic, architectural, LLM-as-Judge, and human-in-the-loop. In Codex, invoke with $adev:eval

Graduated Evaluation Harness

Score implementation quality across four evaluation layers, producing a graduated quality score (0-100) rather than binary pass/fail. Complements /adev:validate with nuanced quality assessment.

Announce at start: "I'm using the adev:eval skill to run the evaluation harness."

Arguments

• --spec <path>: evaluate the implementation of a specific spec (required)
• --layer <N>: run only a specific layer (1-4)
• --configure: interactive setup of eval configuration
• --rubric <path>: use a custom rubric for Layer 3
• --no-infra: skip infrastructure preflight checks (user-only — the agent must never set this flag)

Prerequisites

1. .context-index/ must be initialized.
2. /adev:validate should have passed (eval builds on top of validation, not replaces it).
3. For Layer 2, .context-index/samples/ should have relevant golden samples.
4. Eval configuration lives in .context-index/evals/config.yaml (generated by --configure).

Preflight: Infrastructure Verification

After verifying prerequisites, check whether the spec declares infra_requirements. If so, run the infrastructure preflight before proceeding to evaluation layers.

Layer-aware skip: If --layer 1 or --layer 2 is specified, skip the preflight step (no code execution against external systems in those layers).

--no-infra resolution: Read --no-infra flag from arguments. If not passed, check ADEV_NO_INFRA env var (only exact value 1 activates bypass). Read once at skill entry, convert to options.noInfra. The agent must never set --no-infra or ADEV_NO_INFRA autonomously — if preflight fails, report the failure and wait for user direction.

Plan path resolution: Glob for a .plan.md sibling adjacent to the spec (same directory, same base name). If absent, omit --plan.

Invocation: Run the preflight via the CLI:

adev preflight run --spec <specPath> [--plan <planPath>] [--timeout 10] [--no-infra]

Stdout is a single JSON object — the preflight report (the same shape runPreflight() returns). Exit codes: 0 on PASS or skipped, 2 on FAIL, 1 on argument errors.

If the report has passed === false (exit 2), display the formatted report and block:

Infrastructure Preflight: FAILED

<formatted report output>

Execution blocked. Options:
  1. Fix the issues above and retry
  2. Re-run with --no-infra to bypass (user decision only)

If report.passed === true and report.skipped === true, emit: "Infrastructure preflight skipped (--no-infra)."

If report.passed === true and report.skipped === false, proceed silently.

If lib/infra-preflight.mjs fails to import, block with: "Infrastructure preflight library could not be loaded: . Fix the library before proceeding."

Load Skill Extensions: Load any skill extension instructions before proceeding:

adev skill-ext load --skill eval

If the output is not __NONE__, incorporate it as additional standing instructions that apply to this skill's entire execution. Frame it as: "The following skill extension instructions apply to this invocation (source: installed domain extensions and/or project-level overrides)." If the output is __NONE__, continue normally.

Layer 1: Deterministic Checks (Automated)

Run all quality gates from governance/gates.yaml (or constitution fallback). This mirrors /adev:validate Check 1 but records detailed scores:

• Each gate produces a binary pass/fail
• Overall Layer 1 score = (gates passed / total gates) * 25
• Maximum contribution: 25 points

This layer is the floor. If Layer 1 scores 0, the total score cannot exceed 25 regardless of other layers.

Layer 2: Architectural Conformance (Automated)

Compare the implementation against golden samples and constitutional patterns:

1. Pattern consistency (0-10): For each file touched, check if it follows the pattern established by the matching golden sample. Measure: naming conventions, file structure, import organization, error handling style. Score based on deviation count.

2. Boundary compliance (0-5): Run governance/boundaries.yaml patterns. Score based on violations (0 = full score, each violation deducts 1 point).

3. Complexity metrics (0-5): Check file sizes, function lengths, nesting depth against constitution thresholds. Score based on compliance.

4. Test quality (0-5): Assess test files: do they test behavior (not implementation details)? Do they cover error cases? Are assertions meaningful? Score by checklist.

• Overall Layer 2 score = sum of sub-scores (0-25)
• If no golden samples exist, pattern consistency defaults to 5/10 (neutral).

Layer 3: LLM-as-a-Judge (AI-Assessed)

Dispatch a reviewer subagent with a rubric to score code quality on subjective dimensions that cannot be machine-checked:

1. Readability (0-5): Can a new developer understand this code without additional context? Are variable names descriptive? Is the control flow clear?

2. Maintainability (0-5): Is the code easy to modify? Are concerns separated? Are dependencies explicit? Would a change in one area cascade?

3. Spec fidelity (0-5): Does the implementation capture the spirit of the spec, not just the letter? Are edge cases handled gracefully?

4. Idiomatic usage (0-5): Does the code use the framework and language idiomatically? Does it follow community conventions for the declared platform?

5. Error handling (0-5): Are errors handled at the right level? Are error messages useful? Does the happy path degrade gracefully?

Dispatch the reviewer subagent (reasoning tier — read from model_tiers in .context-index/platform-context.yaml; fall back to the hardcoded default in .context-index/specs/cross-cutting/model-routing.md if unset, and log a one-time advisory). Prepend ultrathink as the first word of the subagent prompt to activate extended thinking. Provide:

• The implementation diff (all files changed)

• The Live Spec

• The rubric (default or custom from --rubric)

• Relevant golden samples for comparison

• Instructions to score each dimension 0-5 with a one-sentence justification

• A self-check instruction: "Before finalizing, verify every score has a justification grounded in the actual code, and no score is based on absence of information."

• A return size constraint: "Keep your response under 1,500 tokens. Score each dimension concisely."

• Overall Layer 3 score = sum of sub-scores (0-25)

Layer 4: Human-in-the-Loop Checkpoints (Manual)

Surface items that require human judgment. This layer does not auto-score but presents a checklist:

1. Business logic correctness: Does the implementation match the product intent? (Only the user can judge this.)
2. UX review: If the task includes UI changes, do they meet design expectations?
3. Security review: For auth, data handling, or API changes, has a human verified the approach?
4. Performance impact: For high-traffic paths, has the performance impact been considered?

Present the checklist to the user. User marks each item as: PASS (5 points), ACCEPTABLE (3 points), NEEDS_WORK (0 points).

• Overall Layer 4 score = (sum of ratings / max possible) * 25
• If user skips Layer 4, it contributes 0 and the max score is 75.

Scoring

Total quality score = Layer 1 + Layer 2 + Layer 3 + Layer 4 (0-100)

Score	Grade	Interpretation
90-100	A	Excellent. Exemplary implementation.
75-89	B	Good. Minor improvements possible.
60-74	C	Acceptable. Notable gaps in quality.
40-59	D	Below standard. Significant issues.
0-39	F	Failing. Major rework needed.

Report Format

Persona adaptation: The report written to disk always uses the full format below. The chat summary presented to the user should follow the active persona's output rules.

Write to .context-index/evals/<spec-slug>-eval.md:

# Evaluation Report: <Spec Title>

> **Date:** YYYY-MM-DD
> **Spec:** <path>
> **Overall Score:** N/100 (Grade: A-F)

## Layer 1: Deterministic Checks — N/25
- Tests: PASS/FAIL
- Lint: PASS/FAIL
- Typecheck: PASS/FAIL

## Layer 2: Architectural Conformance — N/25
- Pattern consistency: N/10
- Boundary compliance: N/5
- Complexity metrics: N/5
- Test quality: N/5

## Layer 3: LLM-as-a-Judge — N/25
- Readability: N/5 — <justification>
- Maintainability: N/5 — <justification>
- Spec fidelity: N/5 — <justification>
- Idiomatic usage: N/5 — <justification>
- Error handling: N/5 — <justification>

## Layer 4: HITL Checkpoints — N/25
- Business logic: PASS/ACCEPTABLE/NEEDS_WORK
- UX review: PASS/ACCEPTABLE/NEEDS_WORK/SKIPPED
- Security review: PASS/ACCEPTABLE/NEEDS_WORK/SKIPPED
- Performance: PASS/ACCEPTABLE/NEEDS_WORK/SKIPPED

## Trend
<comparison to previous evals for this module, if any>

Configuration (--configure)

Interactive setup that creates .context-index/evals/config.yaml:

# Eval harness configuration
layers:
  deterministic: true       # Layer 1 always on
  architectural: true       # Layer 2
  llm_judge: true           # Layer 3
  hitl: false               # Layer 4 (opt-in)

thresholds:
  minimum_score: 60         # Below this triggers a warning
  exemplary_score: 90       # Above this flags as golden sample candidate

rubric: default             # Or path to custom rubric file

Integration

• /adev:validate is the prerequisite (pass/fail gate). /adev:eval adds graduated scoring on top.
• /adev:retro reads eval scores for quality trend analysis across sprints
• /adev:sample candidates can be identified from A-grade evaluations
• /adev:hygiene can include eval score trends in its drift report