eval-run

name	eval-run
description	Execute an evaluation against test cases (skill or prompt mode), score with judges, and report results. Requires eval.yaml (generated by /eval-analyze). Use when the user wants to test a skill, run eval, benchmark, compare models, detect regressions, check skill quality, or verify changes didn't break anything. Triggers on "run eval", "test the skill", "evaluate", "benchmark", "check for regressions", "how does my skill perform", "score the skill", "run the tests", "run my evals", "compare against baseline", "did I break anything", "test my changes". Also called by /eval-optimize for automated iterations.
user-invocable	true
allowed-tools	Read, Write, Edit, Bash, Glob, Grep, Agent, Skill, AskUserQuestion

You are an evaluation executor. You run a skill against test cases, score the outputs with judges, and report results. You orchestrate by calling scripts — never duplicate their work.

For the full data flow (dataset → workspace → execution → collection → scoring), see ${CLAUDE_SKILL_DIR}/references/data-pipeline.md. For tool interception mechanics, see ${CLAUDE_SKILL_DIR}/references/tool-interception.md.

Step 0: Parse Arguments and Load Config

Parse $ARGUMENTS:

Argument	Required	Default	Description
--config <path>	no	auto-discover	Path to eval config
--model <model>	no	models.skill from config	Skill model. Required if models.skill is unset in eval.yaml.
--subagent-model <model>	no	models.subagent → falls back to skill model	Model for subagents (e.g., claude-sonnet-4-6 while main is claude-opus-4-7)
--skill <name>	no	from config	Override the skill to test
--run-id <id>	no	YYYY-MM-DD-<model>	Identifier for this run
--cases <id> [<id> ...]	no	all cases	Exact case IDs to run
--baseline <run-id>	no	—	Previous run to compare against
--no-llm-judges	no	false	Skip LLM judges (prompt, prompt_file, LLM builtins, agent). Run deterministic judges (check, Python builtins, external code).
--gold	no	false	Save outputs as gold references after run
--effort <level>	no	runner.effort from config	Agent reasoning effort (claude-code or codex)
--runner <type>	no	local	local (default Steps 1–8) or harbor (containerized — skips to Harbor runner section)
--env <name>	no	kubernetes	Harbor execution environment: podman, kubernetes, openshift (only with --runner harbor)
--mount <source:target[:ro or rw]>	no	—	Repeatable Podman bind mount; defaults to read-only (only with --runner harbor)
--cpus <n> / --memory-mb <MiB>	no	Harbor defaults	Hard resource limits per Harbor environment

If --runner harbor: after config discovery, skip to the Harbor runner section below. Steps 2–6 are replaced by one run.py call.

Config Discovery

If --config was explicitly provided, use that path directly.

Otherwise, auto-discover eval configs:

python3 ${CLAUDE_SKILL_DIR}/../../scripts/discover.py

• 1 config found: auto-select it as <config>
• Multiple configs found: present the list and ask the user which to run
• No configs found: proceed to the bootstrap flow below

After selecting a config, read its skill field to set <eval-name> (used in $AGENT_EVAL_RUNS_DIR/<eval-name>/<id> paths below).

Check if the resolved config file exists:

test -f <config> && echo "CONFIG_EXISTS" || echo "NO_CONFIG"

If config is missing: bootstrap via /eval-analyze (invoked through the Skill tool) — if skills/ contains skills, suggest /eval-analyze --skill <name>; otherwise /eval-analyze --prompt <path> (prompt mode).

Once config exists, read it to understand the eval setup — the skill under test, runner, dataset, outputs, judges, models, and any tool interception. The downstream scripts read the same config; you don't need to pass these fields through, just confirm they're present and warn the user about anything missing or surprising.

If inputs.tools has entries but the skill uses AskUserQuestion or external APIs, verify the handlers cover those tools. Warn the user if a tool the skill uses isn't intercepted — headless execution may hang.

Persist parsed flags:

mkdir -p tmp ${AGENT_EVAL_RUNS_DIR:-eval/runs}
python3 ${CLAUDE_SKILL_DIR}/scripts/agent_eval/state.py init tmp/eval-config.yaml \
  model=<model> skill=<skill> run_id=<id> baseline=<baseline> \
  gold=<true/false> no_llm_judges=<true/false>

Step 1: Find Dataset

Read dataset.path from eval.yaml. Verify the directory exists and contains at least one case subdirectory:

ls <dataset_path>/ | head -20

If --cases was specified, pass the IDs to workspace.py as --cases <id> <id> ....

If no cases found, stop and tell the user clearly:

• What path was checked
• That they need test cases there
• Suggest running /eval-dataset to generate test cases, or /eval-analyze --update to reconfigure the dataset path

Step 2: Preflight Check

Before setting up the workspace, verify the project's artifact directories are clean. Skills write to the project directory (not the workspace), so stale artifacts from previous runs contaminate results — wrong IDs, stale run reports, inflated file counts.

python3 ${CLAUDE_SKILL_DIR}/scripts/preflight.py \
  --config <config> \
  [--run-id <id>] \
  [--baseline <baseline-id>]

The script checks tmp/ state files, whether $AGENT_EVAL_RUNS_DIR/<eval-name>/<id> already has results from a previous run, and (if --baseline is given) that the baseline run-id exists under the same eval-name directory.

• CLEAN: proceed to workspace setup.
• DIRTY: report the findings and ask the user to choose — force clean (preflight.py --clean --force, then proceed), change run-id (append a suffix like -opus-v2, then re-run preflight with --clean on the new id to avoid overwriting prior results), or abort for manual cleanup.
• MISSING_BASELINE (exit 2): the --baseline run-id wasn't found under $AGENT_EVAL_RUNS_DIR/<eval-name>/. The script lists nearby run-ids — confirm the correct one with the user before retrying.

Step 3: Prepare Workspace

Create an isolated workspace with the test cases and output directories:

python3 ${CLAUDE_SKILL_DIR}/scripts/workspace.py \
  --config <config> \
  --run-id <id> \
  [--cases <id> [<id> ...]]

The script prints WORKSPACE: <path>, CASES: <count>, BATCH: <path>. Report these to the user. If inputs.tools is configured, it also prints HOOKS: N tool interceptors configured.

If the case count is 0, stop — the filter matched nothing.

Step 3a: Resolve Tool Interception (if inputs.tools configured)

If eval.yaml has inputs.tools entries, this step is mandatory. workspace.py emits a skeleton in tool_handlers.yaml; you must resolve each handler's prompt into concrete runtime checks (input_filters, env_checks, case_overrides). Do not skip this even when the eval.yaml is unchanged — the workspace is created fresh each time.

Read ${CLAUDE_SKILL_DIR}/references/tool-interception.md for the full format, field reference, and resolution examples. Then read <workspace>/tool_handlers.yaml and for each handler:

1. Identify type: AskUserQuestion, Bash, or MCP tool
2. Add required fields: input_filters for Bash, env_checks for services, case_overrides for deterministic answers
3. Verify: every Bash handler has input_filters — without them the handler is non-functional

Write the resolved handlers back to tool_handlers.yaml.

Step 4: Execute

Run the eval headlessly against test cases. execute.py auto-detects the execution mode from the config: skill mode (execution.skill + arguments) invokes the skill, while prompt mode (execution.prompt) sends the prompt directly with no skill wrapper. For how the mode is selected and how templates are resolved, see ${CLAUDE_SKILL_DIR}/references/execution-modes.md.

In case mode (default), execute.py runs once per case with case-specific arguments and workspace — each case gets its own stdout.log and subagent transcripts. In batch mode, all cases run in a single invocation via batch.yaml.

If hooks: is configured in eval.yaml, execute.py automatically runs lifecycle hooks at the appropriate points: before_all before any case executes, before_each/after_each around each case execution, and after_all after all cases complete (guaranteed, even on failure). Hook logs are written to $AGENT_EVAL_RUNS_DIR/<id>/hooks/.

python3 ${CLAUDE_SKILL_DIR}/scripts/execute.py \
  --config <config> \
  --workspace <workspace_path> \
  --skill <skill_name> \
  --skill-args "<skill arguments>" \
  --model <model> \
  --output $AGENT_EVAL_RUNS_DIR/<eval-name>/<id> \
  --run-id <id> \
  [--agent <runner>] \
  [--subagent-model <model>] \
  [--mlflow-experiment <name>] \
  [--effort <level>] \
  [--parallelism <n>]

Launch with run_in_background: true and do NOT redirect its output (no >, |, tee, 2>&1) — a redirect diverts the stream Claude Code's background viewer reads and leaves it blank. You don't need one: execute.py mirrors its live console to <output_dir>/console.log.

When it finishes, read run_result.json — if exit_code is non-zero, report the failure and stop. If stderr.log ends with a bg-wait-ceiling ERROR:, fix the skill's turn discipline if it should have awaited its agents; raise CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS via runner.env: only for intentionally long-running background work. For session-lifecycle details, polling patterns, problem detection, and CLI fallbacks, see references/execution-monitoring.md.

Step 5: Collect Artifacts

Distribute workspace outputs into per-case directories so judges can score each case independently:

python3 ${CLAUDE_SKILL_DIR}/scripts/collect.py \
  --config <config> \
  --workspace <workspace_path> \
  --output $AGENT_EVAL_RUNS_DIR/<eval-name>/<id>

Read the collection summary (JSON file — read it with cat or jq, not state.py):

cat $AGENT_EVAL_RUNS_DIR/<eval-name>/<id>/collection.json

Report per-case counts. If any case has 0 artifacts, warn — the skill may not have produced output for that case.

Step 6: Score

Run all configured judges against the collected outputs. Five judge types: builtin, inline check, LLM (prompt/prompt_file/llm_rubric), external module/function, and agent (a tool-using judge run through the runner; writes output/score.json). All take optional arguments:. See references/data-pipeline.md.

If --no-llm-judges was specified, skip judges that make LLM API calls (prompt, prompt_file, LLM builtins, and agent judges). Run deterministic judges only (check, Python builtins, external code).

python3 ${CLAUDE_SKILL_DIR}/scripts/score.py judges \
  --run-id <id> \
  --config <config> \
  --workspace <workspace_path> \
  --model <model> \
  [--no-llm-judges]   # add when the run requested skipping model-calling judges

If hooks.before_scoring is configured in eval.yaml, score.py runs those hooks before judge execution. Pass --workspace and --model so hook environment variables are populated.

Judges receive the case outputs record — file contents (outputs["files"], <dir>_content), execution metadata, tool_calls, logs, and annotations (parsed annotations.yaml, for outcome-aware scoring). Full field reference: references/data-pipeline.md.

If --baseline was specified, also run pairwise comparison:

python3 ${CLAUDE_SKILL_DIR}/scripts/score.py pairwise \
  --run-id <id> \
  --baseline <baseline_id> \
  --config <config>

Read the full results:

python3 ${CLAUDE_SKILL_DIR}/scripts/agent_eval/state.py read $AGENT_EVAL_RUNS_DIR/<eval-name>/<id>/summary.yaml

summary.yaml has three sections: judges (per-judge mean and pass_rate), per_case (per-case {value, rationale} per judge), and pairwise (only if --baseline was used: run_a, run_b, wins_a, wins_b, ties).

Step 7: Interpret and Report

Read the summary and analyze the results. Read ${CLAUDE_SKILL_DIR}/prompts/analyze-results.md for the full analysis framework and output format. Lead with Recommendation (self-contained — many readers only read this section). Save the analysis to $AGENT_EVAL_RUNS_DIR/<eval-name>/<id>/analysis.md with YAML frontmatter (agent, model, date) — see the prompt file for the template.

Generate HTML report:

python3 ${CLAUDE_SKILL_DIR}/scripts/report.py \
  --run-id <id> \
  --config <config> \
  [--baseline <baseline_id>] \
  --open

Tell the user the report is at $AGENT_EVAL_RUNS_DIR/<eval-name>/<id>/report.html.

If --gold flag: After scoring, copy collected artifacts to dataset case dirs as reference files. Report which cases were saved.

Suggest next steps (include --config <config> if a non-default config was used):

• /eval-review --run-id <id> for interactive human review of the results
• /eval-optimize --model <model> for automated improvement based on failures
• /eval-mlflow --run-id <id> to log results to MLflow

Step 8: Log to MLflow (optional)

If mlflow.experiment is configured in eval.yaml:

Use the Skill tool to invoke /eval-mlflow --action log-results --run-id <id> --config <config>

Harbor runner (--runner harbor)

When --runner harbor is specified, skip Steps 2–6 and call run.py instead — it handles task generation (or reuse), harbor run, per-case judging (in-container), result mapping, and report generation in one call:

# Run under the eval-harness venv interpreter so compiled deps load with the
# correct ABI and the process is never re-exec'd mid-run. Falls back to system
# python3 if the venv is absent. The venv holds only third-party deps, so the
# plugin root (source of the agent_eval package) must be on PYTHONPATH.
PLUGIN_ROOT="${CLAUDE_PLUGIN_ROOT:-$CLAUDE_SKILL_DIR/../..}"
VENV_PY="$PLUGIN_ROOT/.eval-venv/bin/python3"
[ -x "$VENV_PY" ] || VENV_PY=python3

PYTHONPATH="$PLUGIN_ROOT:$(pwd)${PYTHONPATH:+:$PYTHONPATH}" "$VENV_PY" -m agent_eval.harbor.run \
    --config <config> --model <model> \
    --output $AGENT_EVAL_RUNS_DIR/<eval-name>/<run-id> \
    --tasks-dir <tasks-dir> --jobs-dir <tmp-jobs> \
    [--image <image>] [--agent <agent>] [--n-concurrent N] \
    [--env kubernetes] [--mount <source:target:ro>] [--no-llm-judges] \
    [--cpus N] [--memory-mb MiB]

For runner.type: codex, run.py selects Harbor's Codex agent, passes runner.effort as reasoning_effort, and passes every runner.plugin_dirs entry's complete manifest-declared skill roots. Do not narrow an orchestrator to its primary skill directory; sibling dependencies must remain discoverable. Use --mount with Podman for large datasets instead of copying them into each task. Kubernetes requires cluster-visible storage rather than host paths.

Cluster-specific config (namespace, credentials secret) is read from a .env file in the project root. Create it with AGENT_EVAL_K8S_NAMESPACE and AGENT_EVAL_K8S_CREDENTIALS_SECRET — run.py loads it automatically.

Tasks come from /eval-dataset (which emits Harbor task packages via scripts/harbor.py). If --tasks-dir already has them, run.py reuses them; if empty, it generates on the fly (needs --image). The output is a standard run_result.json + summary.yaml + report.html — then continue with Step 6 pairwise (if --baseline) and Step 8 (MLflow) as normal. See deploy/harbor/README.md for image build, credentials, and environment setup.

--no-llm-judges is a generation-only option. It requires an image and either an empty tasks directory or --regenerate; generated packages are stamped as deterministic-only and cannot later be reused for a full judge run.

EvalHub runner (--runner evalhub)

When --runner evalhub is specified, skip Steps 2–8 and call the EvalHub runner instead — it creates ConfigMaps, submits a job to EvalHub, polls for completion, and maps the results back:

PLUGIN_ROOT="${CLAUDE_PLUGIN_ROOT:-$CLAUDE_SKILL_DIR/../..}"
VENV_PY="$PLUGIN_ROOT/.eval-venv/bin/python3"
[ -x "$VENV_PY" ] || VENV_PY=python3

PYTHONPATH="$PLUGIN_ROOT:$(pwd)${PYTHONPATH:+:$PYTHONPATH}" "$VENV_PY" -m agent_eval.evalhub.runner \
    --config <config> --model <model> \
    --output $AGENT_EVAL_RUNS_DIR/<eval-name>/<run-id> \
    [--evalhub-url <url>] [--namespace <ns>] [--project-dir <path>]

The runner creates K8s ConfigMaps for the eval config and project resources (via k8s_resources), submits the job to EvalHub (which creates a Job pod running the adapter in-process), polls until completion, and maps the results into the standard summary.yaml + report.html. No image rebuild needed — ConfigMaps carry the project-specific content.

Rules

• Never read large artifact files into your context — delegate content analysis to agents. The summary.yaml has everything you need for reporting.
• Persist state at every step — use python3 ${CLAUDE_SKILL_DIR}/scripts/agent_eval/state.py so flags and results survive context compression.
• Report progress at each step so the user knows what's happening and how long it's taking.
• Fail fast — if execution fails, report it immediately. Don't continue to scoring with no artifacts.
• Be decisive in analysis — the user wants to know what's wrong and what to do about it, not a list of possibilities.

$ARGUMENTS