agentv-eval-review

name	agentv-eval-review
description	Use when reviewing eval YAML files for quality issues, linting eval files before committing, checking eval schema compliance, or when asked to "review these evals", "check eval quality", "lint eval files", or "validate eval structure". Do NOT use for writing evals (use agentv-eval-writer) or running evals (use agentv-bench).

Eval Review

Overview

Lint and review AgentV eval YAML files for structural issues, schema compliance, and quality problems. Runs deterministic checks via script, then applies LLM judgment for semantic issues the script cannot catch.

Process

Step 1: Run the linter

Execute scripts/lint_eval.py against the target eval files:

python scripts/lint_eval.py <path-to-evals-dir-or-file> --json

The script checks:

• .eval.yaml extension
• description field present
• Each test has id, input, and at least one of criteria/expected_output/assertions
• File paths in type: file use leading /
• assertions blocks present (flags tests relying solely on expected_output)
• expected_output prose detection (flags "The agent should..." patterns)
• Repeated file inputs across tests (recommends top-level input)
• Naming prefix consistency across eval files in same directory

Step 2: Review script output

Report the script findings grouped by severity (error > warning > info). For each finding, include the file path and a concrete fix.

Step 3: Semantic review (LLM judgment)

The script catches structural issues but cannot assess:

• Factual accuracy — Do tool/command names in expected_output match what the skill documents?
• Coverage gaps — Are important edge cases missing?
• Assertion discriminability — Would assertions pass for both good and bad output?
• Cross-file consistency — Do output filenames match across evals and skills?

Read the relevant SKILL.md files and cross-check against the eval content for these issues.

Skill Resources

• scripts/lint_eval.py — Deterministic eval linter (Python 3.11+, stdlib only)