eval-skills

name	eval-skills
description	AI Agent Skill unit testing framework. A framework-agnostic toolkit for discovering, scaffolding, selecting, evaluating, and reporting on AI skills. Use this skill to assess skill quality before production, compare candidate skills on the same benchmark, enforce quality gates in CI/CD, and generate human-readable evaluation reports.
version	0.1.0

eval-skills

AI Agent Skill unit testing framework — a framework-agnostic toolkit for discovering, scaffolding, selecting, evaluating, and reporting on AI skills.

This skill fills the L1 (Skill Unit Test) gap that LangSmith / DeepEval leave open: while those platforms focus on agent-level and trajectory-level evaluation (L2-L3), eval-skills targets the individual skill level, ensuring each building block meets quality standards before it ever enters an agent pipeline.

When to Use This Skill

• Before deploying a new skill to production — run eval to verify it meets your quality gate.
• When choosing between multiple candidate skills — run select to rank them on the same benchmark.
• When a skill is upgraded — run report diff to detect regressions.
• In CI/CD — use --exit-on-fail to block merges that degrade skill quality.
• When bootstrapping a new skill — run create to generate a ready-to-fill skeleton.

Capabilities

1. Find Skills

Search for existing skills by keyword, tag, or adapter type.

eval-skills find \
  --query "web search" \
  --tag retrieval api \
  --adapter http \
  --min-completion 0.8 \
  --skills-dir ./skills \
  --limit 10

Option	Description	Default
-q, --query <string>	Keyword search (matches name, description, tags)	—
-t, --tag <tags...>	Filter by tags (intersection: skill must have ALL specified tags)	—
-a, --adapter <type>	Filter by adapter type (http, subprocess, mcp)	—
--min-completion <rate>	Minimum historical completion rate (0.0 ~ 1.0)	—
--skills-dir <dir>	Directory to scan for skill.json files	./skills
--limit <n>	Maximum number of results	20

Results are ranked by search relevance (when --query is provided) or by historical completion rate (descending).

2. Create Skills

Generate a skill skeleton from a template to bootstrap development.

eval-skills create \
  --name my_api_skill \
  --from-template http_request \
  --output-dir ./skills \
  --description "Fetches weather data from OpenWeather API"

Option	Description	Default
--name <name>	Required. Skill name	—
--from-template <tpl>	Template type: http_request, python_script, mcp_tool	http_request
--output-dir <dir>	Output directory	./skills
--description <text>	Human-readable description embedded in skill.json	—

Generated file structure:

skills/my_api_skill/
  skill.json            # Skill metadata (id, schemas, adapter config)
  adapter.config.json   # Adapter-specific configuration
  tests/
    basic.eval.json     # A starter benchmark with one sample task
  skill.py              # (python_script template only) JSON-RPC entrypoint

3. Evaluate Skills

Run benchmark evaluations against one or more skills. This is the core command.

eval-skills eval \
  --skills ./skills/calculator/skill.json ./skills/search/ \
  --benchmark coding-easy \
  --concurrency 4 \
  --timeout 30000 \
  --retries 2 \
  --runs 3 \
  --evaluator exact \
  --format json markdown html \
  --output-dir ./reports \
  --exit-on-fail --min-completion 0.8 \
  --store ./eval-skills.db

Option	Description	Default
--skills <paths...>	Required. Skill file(s) or directory(ies)	—
--benchmark <id|path>	Built-in benchmark ID or path to benchmark.json	coding-easy
--tasks <file>	Custom tasks JSON file (replaces benchmark)	—
--concurrency <n>	Number of parallel task executions	4
--timeout <ms>	Per-task timeout in milliseconds	30000
--retries <n>	Retry count on task failure (with incremental backoff)	0
--runs <n>	Repeat evaluation N times for consistency scoring	1
--evaluator <type>	Default scorer type (see Scorer Types below)	exact
--format <formats...>	Output formats: json, markdown, html	json markdown
--output-dir <dir>	Report output directory	./reports
--exit-on-fail	Exit with code 1 if any skill falls below threshold	disabled
--min-completion <rate>	Threshold for --exit-on-fail	0.7
--dry-run	Validate configuration only; do not execute tasks	disabled
--benchmarks-dir <dir>	Directory containing built-in benchmarks	./benchmarks
--store <path>	SQLite database path for persistent result storage	./eval-skills.db
-c, --config <path>	Path to eval-skills.config.yaml	auto-detected

Evaluation flow:

1. Load skills from --skills paths (supports both single skill.json and directories)
2. Load benchmark tasks from --benchmark or --tasks
3. Build the cartesian product: skills x tasks x runs
4. Execute all task items concurrently (controlled by --concurrency, with timeout and retry)
5. Score each result using the appropriate scorer
6. Aggregate into SkillCompletionReport per skill
7. Write reports to --output-dir

4. Select Skills

Filter and rank skills based on evaluation reports using a multi-dimensional strategy.

eval-skills select \
  --from ./skills \
  --reports ./reports/eval-result.json \
  --strategy ./strategy.yaml \
  --min-completion 0.8 \
  --top-k 5 \
  --output ./selected.json

Option	Description	Default
--from <path>	Required. Candidate skills directory or JSON file	—
--reports <file>	Evaluation reports JSON file	—
--strategy <file>	SelectStrategy YAML/JSON file	built-in default
--min-completion <rate>	Override minimum completion rate filter	—
--top-k <n>	Return only the top K results	all
--output <file>	Write selected skills to file	stdout

Selection pipeline: Filter (by completion rate, error rate, latency, adapter type, required tags) -> Score -> Rank (by compositeScore, completionRate, latency, or tokenCost) -> TopK

Example strategy.yaml:

filters:
  minCompletionRate: 0.8
  maxErrorRate: 0.1
  maxLatencyP95Ms: 5000
  adapterTypes: [http, subprocess]
  requiredTags: [production-ready]
sortBy: compositeScore
order: desc
topK: 5

5. Run Pipeline

Execute the full end-to-end pipeline: Find -> Eval -> Select -> Report in a single command.

eval-skills run \
  --query "math" \
  --benchmark coding-easy \
  --skills-dir ./skills \
  --top-k 3 \
  --min-completion 0.7 \
  --format json markdown \
  --output-dir ./reports

This command automates the entire process:

1. Find — scans --skills-dir and optionally filters by --query
2. Eval — evaluates all candidate skills against --benchmark
3. Select — filters and ranks results using --min-completion, --top-k, and optional --strategy
4. Report — generates output files in all requested --formats

6. Generate & Compare Reports

Convert report format

eval-skills report convert \
  --input ./reports/eval-result.json \
  --format html \
  --output ./reports/eval-result.html

Supported output formats: markdown, html.

Diff two reports (regression detection)

eval-skills report diff \
  ./reports/v1.json ./reports/v2.json \
  --label-a "v1.0" --label-b "v2.0" \
  --output ./reports/diff.md

Generates a side-by-side delta table per skill showing changes in completion rate, error rate, P95 latency, and composite score with directional arrows.

7. Initialize Project

eval-skills init --dir .

Creates the project scaffold:

• eval-skills.config.yaml — global configuration
• skills/ — directory for skill definitions
• benchmarks/ — directory for benchmark files
• reports/ — directory for evaluation output

8. Manage Configuration

# List all current configuration values
eval-skills config list

# Get a specific value (supports dot notation)
eval-skills config get llm.model

# Set a value (persisted to ~/.eval-skills/config.yaml)
eval-skills config set concurrency 8
eval-skills config set llm.model gpt-4o
eval-skills config set llm.temperature 0

Configuration is resolved in priority order:

1. CLI flags (highest priority)
2. eval-skills.config.yaml in current directory
3. ~/.eval-skills/config.yaml
4. Built-in defaults (concurrency: 4, timeoutMs: 30000, outputDir: ./reports)

Scorer Types

Each task in a benchmark specifies an evaluator type. The scorer compares the skill's actual output against the expected output.

Type	Aliases	Description	Score Range
exact_match	exact	Strict equality comparison. Supports caseSensitive option.	0 or 1
contains	—	Checks for the presence of all specified keywords in the output. Partial credit: matched_keywords / total_keywords.	0.0 ~ 1.0
json_schema	schema	Validates output against a JSON Schema (using Ajv).	0 or 1
llm_judge	—	Sends the output + expected rubric to an LLM (configurable model) for quality rating.	0.0 ~ 1.0
custom	—	Loads a custom scorer from expectedOutput.customScorerPath.	0.0 ~ 1.0

Evaluation Metrics

Every evaluation produces a SkillCompletionReport with these metrics:

Metric	Description	Formula
Completion Rate	Fraction of tasks that passed	pass_count / total_count
Partial Score	Mean score across all tasks	mean(task_scores)
Error Rate	Fraction of tasks that errored or timed out	(error_count + timeout_count) / total_count
Consistency Score	Stability across multiple runs (requires --runs >= 2)	1 - stddev(per_run_completion_rates)
P50 / P95 / P99 Latency	Response time percentiles	Sorted percentile of latencyMs
Composite Score	Weighted overall quality score	0.5 * CR + 0.2 * (1 - latP95_norm) + 0.3 * (1 - ER)

Built-in Benchmarks

ID	Domain	Tasks	Scoring	Description
coding-easy	coding	20	mean / exact_match	Math expressions, string reversal, palindrome detection
skill-quality	tool-use	5	mean / contains	Metadata completeness, description quality, structure checks
web-search-basic	web	8	mean / contains + schema	Factual queries, keyword verification, structured output validation
gaia-v1	general	—	mean	Placeholder for GAIA benchmark Level 1 tasks
toolbench-lite	tool-use	—	mean	Placeholder for ToolBench single-tool scenarios

Custom Benchmark

Create a benchmark.json file:

{
  "id": "my-benchmark",
  "name": "My Custom Benchmark",
  "version": "1.0.0",
  "domain": "general",
  "scoringMethod": "mean",
  "maxLatencyMs": 30000,
  "metadata": { "source": "internal", "lastUpdated": "2026-02-28" },
  "tasks": [
    {
      "id": "task_001",
      "description": "Test basic addition",
      "inputData": { "expression": "2+3" },
      "expectedOutput": { "type": "exact", "value": "5" },
      "evaluator": { "type": "exact" },
      "timeoutMs": 10000,
      "tags": ["math"]
    },
    {
      "id": "task_002",
      "description": "Test keyword presence",
      "inputData": { "query": "TypeScript" },
      "expectedOutput": { "type": "contains", "keywords": ["JavaScript", "Microsoft"] },
      "evaluator": { "type": "contains", "caseSensitive": false },
      "timeoutMs": 15000,
      "tags": ["search"]
    }
  ]
}

eval-skills eval --skills ./my-skill/ --benchmark ./my-benchmark.json

Adapter Types

Skills communicate through adapters. The adapter type is specified in skill.json via adapterType.

Adapter	Protocol	How it works	Key config
http	REST POST	Sends POST { skillId, version, input } to skill.entrypoint. Supports Bearer / API-Key auth via env vars.	baseUrl, authType, authTokenEnvKey
subprocess	JSON-RPC 2.0 over stdin/stdout	Spawns skill.entrypoint (e.g. python3 skill.py), writes JSON-RPC request to stdin, reads response from stdout.	command, args
mcp	MCP Protocol	(Phase 2) Native Model Context Protocol integration via @modelcontextprotocol/sdk.	—

Workflow Examples

Evaluating a Single Skill

# 1. Create a skill skeleton
eval-skills create --name my_calc --from-template python_script

# 2. Implement your logic in skills/my_calc/skill.py

# 3. Run evaluation against the coding-easy benchmark
eval-skills eval \
  --skills ./skills/my_calc/skill.json \
  --benchmark coding-easy \
  --runs 3 \
  --format json markdown

# 4. Review the report
cat ./reports/eval-result-*.md

Comparing Multiple Candidate Skills

# 1. Discover candidates
eval-skills find --query "weather" --skills-dir ./skills

# 2. Evaluate all candidates on the same benchmark
eval-skills eval \
  --skills ./skills/weather_v1 ./skills/weather_v2 ./skills/weather_v3 \
  --benchmark web-search-basic \
  --runs 3

# 3. Select the best
eval-skills select \
  --from ./skills \
  --reports ./reports/eval-result-*.json \
  --min-completion 0.8 \
  --top-k 2

# 4. Compare two versions
eval-skills report diff \
  ./reports/v1.json ./reports/v2.json \
  --label-a "weather_v1" --label-b "weather_v2"

Full Pipeline (One Command)

eval-skills run \
  --skills-dir ./skills \
  --benchmark coding-easy \
  --top-k 3 \
  --min-completion 0.7 \
  --format json markdown html \
  --output-dir ./reports

CI/CD Quality Gate

# In your CI pipeline — fail the build if completion rate drops below 80%
eval-skills eval \
  --skills ./skills/production_skill \
  --benchmark coding-easy \
  --exit-on-fail \
  --min-completion 0.8 \
  --format json

Regression Detection

# Compare today's evaluation against the baseline
eval-skills report diff \
  ./reports/baseline.json ./reports/latest.json \
  --label-a "baseline" --label-b "latest" \
  --output ./reports/regression-check.md

Best Practices

1. Always use --runs 3 or more when evaluating for production decisions. Single-run results can be noisy; the consistency score captures stability across runs.

2. Use --exit-on-fail in CI/CD pipelines to enforce quality gates. Set --min-completion to your acceptable threshold (recommended: 0.8 for production skills).

3. Create domain-specific custom benchmarks rather than relying solely on built-in ones. Your custom benchmark should reflect real-world inputs your skill will encounter.

4. Use report diff after every skill upgrade to catch regressions early. Compare the new evaluation against a saved baseline report.

5. Use --dry-run before long evaluations to validate your configuration (skill paths, benchmark resolution, task count) without actually executing tasks.

6. Persist results with --store to track skill quality over time. The SQLite store enables historical trend queries.

7. Start with --concurrency 1 when debugging a failing skill, then increase for production benchmarking.

8. Tag your benchmark tasks to enable per-category analysis (e.g., filter by math, string, edge-case).

Skill JSON Schema

Every skill must provide a skill.json that conforms to this structure:

{
  "id": "my_skill_v1",
  "name": "My Skill",
  "version": "1.0.0",
  "description": "Does something useful",
  "tags": ["utility", "math"],
  "inputSchema": {
    "type": "object",
    "properties": { "query": { "type": "string" } },
    "required": ["query"]
  },
  "outputSchema": {
    "type": "object",
    "properties": { "result": { "type": "string" } }
  },
  "adapterType": "subprocess",
  "entrypoint": "python3 skill.py",
  "metadata": {
    "author": "Your Name",
    "license": "MIT",
    "homepage": "https://github.com/you/my-skill"
  }
}

Validation rules:

• id: lowercase alphanumeric with _ or -, non-empty
• version: semver format (X.Y.Z)
• adapterType: one of http, subprocess, mcp, langchain, custom
• entrypoint: non-empty string (URL for http, command for subprocess)

Global Options

These options are available on all commands:

Option	Description
-c, --config <path>	Path to configuration file
--json	JSON output format (CI-friendly)
--no-color	Disable colored output
-v, --verbose	Verbose logging
--version	Show version
-h, --help	Show help