Create, run, and manage AgentV evaluations for AI agents and skills using the AgentV CLI and AgentEvals standard EVAL.yaml format. Use this skill whenever the user wants to write evaluation files for AI agents, run evals with agentv CLI, convert existing test cases to EVAL.yaml, set up eval targets, understand the AgentEvals specification, debug failing evaluations, integrate evals into CI/CD pipelines, or compare agent runs with `agentv compare`. Also use this when the user mentions EVAL.yaml, agentevals, agentv, or wants to evaluate skill quality with a declarative format.
Create, run, and manage AgentV evaluations for AI agents and skills using the AgentV CLI and AgentEvals standard EVAL.yaml format. Use this skill whenever the user wants to write evaluation files for AI agents, run evals with agentv CLI, convert existing test cases to EVAL.yaml, set up eval targets, understand the AgentEvals specification, debug failing evaluations, integrate evals into CI/CD pipelines, or compare agent runs with `agentv compare`. Also use this when the user mentions EVAL.yaml, agentevals, agentv, or wants to evaluate skill quality with a declarative format.
license
MIT
compatibility
Requires Node.js 18+ with `agentv` globally installed (`npm install -g agentv`). Supports any agent target — Claude, Codex, Copilot, local CLI scripts, or OpenAI-compatible providers.
metadata
{"author":"Tyler Kendrick","version":"0.1.0"}
AgentV Skill
Use this skill to author, run, and manage evaluations for AI agents and skills using the AgentV CLI and the AgentEvals declarative YAML standard.
Read references/eval-yaml-schema.md for the complete EVAL.yaml schema reference before authoring new eval files.
Read references/targets.md for target configuration options.
When to use this skill
The user wants to write EVAL.yaml evaluation files for a skill or agent.
The user wants to run evals with agentv eval.
The user wants to convert existing test cases (e.g., evals.json) to EVAL.yaml format.
The user wants to understand the AgentEvals specification or schema.
The user wants to set up or update .agentv/targets.yaml.
The user wants to run agentv compare to detect regressions across runs.
The user wants to integrate AgentV into a CI/CD pipeline.
The user wants to write LLM judges, rubrics, or code graders for evaluation.
Do not use this skill for general prompt engineering unrelated to evaluations, or for creating and improving agent skill SKILL.md files and their structure.
Quickstart
# Install
npm install -g agentv
# Initialize project
agentv init
# Run an eval
agentv eval evals/my-skill.eval.yaml
# Compare runs
agentv compare .agentv/results/runs/<timestamp>/index.jsonl
# Output formats
agentv eval evals/my.yaml -o report.html # HTML dashboard
agentv eval evals/my.yaml -o results.xml # JUnit XML for CI
agentv eval evals/my.yaml -o results.jsonl # JSONL (default)
Core workflow
Follow this order when creating or updating evals:
Understand the skill/agent — what does it do, what inputs does it accept, what outputs should it produce?
Set up the target — configure .agentv/targets.yaml to point to the agent being evaluated.
Author the EVAL.yaml — write test cases with inputs, expected outputs, and assertions.
Add evaluators — choose deterministic assertions (contains, equals, regex, is-json) or LLM judges for subjective quality.
Run and inspect — agentv eval produces JSONL output; use --output report.html for a visual dashboard.
Iterate — improve the skill based on failing tests and re-run.
Converting evals.json to EVAL.yaml
When migrating an existing evals/evals.json file, map the fields as follows:
evals.json field
EVAL.yaml field
skill_name
name (top-level)
evals[].id
tests[].id
evals[].prompt
tests[].input
evals[].expected_output
tests[].criteria
evals[].assertions[]
tests[].rubrics[] or tests[].assert[]
String-only assertions in evals.json become rubrics (string format) or llm-grader assertions in EVAL.yaml. Preserve both evals.json and EVAL.yaml — the JSON is used by the internal evaluation framework; the YAML is used by AgentV.
Writing judge prompts — use these template variables inside the markdown file:
{{answer}} — the agent's actual response
{{expected_output}} — the expected output from the test case
{{input}} — the original prompt sent to the agent
{{criteria}} — the test case criteria field
Threshold vs. --threshold: The threshold field on an llm-grader assertion is per-assertion (minimum score to pass). The --threshold CLI flag sets a minimum overall suite pass rate for CI gating. These are independent.
Rubrics
Use rubrics when you need weighted or structured grading criteria:
Use top-level assert: to apply evaluators to all test cases:
assert:-type:is-json# All responses must be valid JSON-type:llm-graderprompt:./graders/quality.mdtests:-id:test-1input:...assert:-type:contains# Per-test assertion (merged with suite-level)value:"status"
File references
Test inputs and judge prompts can reference files:
Configure .agentv/targets.yaml to point to your agent. Run agentv init to create a starter file.
# .agentv/targets.yamldefault:type:openaimodel:gpt-4oapi_key:${OPENAI_API_KEY}# Use env vars — never hardcode keysclaude:type:anthropicmodel:claude-opus-4-5api_key:${ANTHROPIC_API_KEY}
Read references/targets.md for all target types (OpenAI, Anthropic, local CLI, HTTP endpoints).
For CI, set the API key as a repository secret and inject it into the environment:
When evaluating skills in this repository, each skill keeps both formats:
skills/my-skill/
└── evals/
├── evals.json # Internal eval framework format
└── EVAL.yaml # AgentV-compatible format
Both files cover the same test cases. evals.json drives the internal skill-creator benchmarking workflow; EVAL.yaml enables agentv eval CLI execution and CI integration.
