zoto-compare-evals

name

zoto-compare-evals

description

Cross-run analysis across two or more eval runs driven by **`report.yml`**. Inputs are resolved per run directory under `{evalsDir}/_runs/<ts>/`; per-backend summaries are nested under **`report.static`** and **`report.llm`**. Emits a flat dataset (runs × cases × dimensions) plus the /canvas hand-off prompt, instructing the host agent to invoke Cursor's built-in /canvas tool. The skill never renders charts itself. Drill-down opens per-case log files. Does not call askQuestion — when a fragment (e.g. `20260503`) matches multiple hourly folders and the Task lacks a disambiguator, emit schema-valid **`needs_user_input`** whose **`options[].label`** lists every full **`{evalsDir}/_runs/<ts>/`** candidate path (`/z-eval-compare` surfaces this without **`askQuestion`**).

Compare Evals

Compare two or more eval runs. The skill does not render any UI; it emits a structured dataset and asks the host agent to route the dataset to Cursor's /canvas tool.

Configuration

Reads .zoto/eval-system/config.yml for evalsDir. Accepts run identifiers as either directory names under {evalsDir}/_runs/ (timestamps such as 20260503051900, or symbolic stamps like run-a / run-b / run-c) or paths that resolve to that run folder.

File layout / reads

Comparison is anchored on each run’s report.yml (merged static + LLM). When flattening datasets, static.yml carries static-backend case rows/outcomes and llm.yml carries LLM case rows/metrics; per-backend rollup numbers also appear under report.static and report.llm inside report.yml. Logs for drill-down: {evalsDir}/_runs/<run>/logs/<case>.log.

Usage

/z-eval-compare <run-1> <run-2> [<run-N>]

Prefer receiving identifiers that resolve to exactly one _runs/ folder. If /z-eval-compare received only a fragment (20260503) and multiple hourly directories match, the command surfaces needs_user_input with {evalsDir}/_runs/... paths — no askQuestion pre-step.

Workflow

Step 1: Resolve runs

For each argument, locate the corresponding run directory (and its report.yml). Prefer report.yml as the canonical comparison input; use sibling static.yml / llm.yml when a dimension needs backend-specific replay.

Matching: Accept directory basenames (20260503121500), relative paths under {evalsDir}/_runs/, or fragments: any _runs/ child whose basename is exactly the token or begins with the token (so 20260503 matches 20260503120000 and 20260503121500).

Ambiguity: If lookup yields multiple _runs/ directories for one argument and the Task prompt did not pin that argument to a single path (exact basename, unequivocal relative path, or other explicit resolution), stop comparison and emit needs_user_input validated against templates/schema/needs-user-input.schema.json. Encode disambiguation as one question per unresolved argument (id: e.g. disambiguate-run-1). Each options[].label MUST be the full run path {evalsDir}/_runs/<matched-basename>/ so every candidate folder is enumerated; options[].id SHOULD be slug-safe (e.g. the basename itself when it fits the slug pattern).

Never call askQuestion; the invoking command relays this payload without prompting.

Step 2: Load and normalise

Parse each report.yml (and pull per-backend totals from report.static / report.llm as needed); include linked rows from static.yml / llm.yml where the flatten step requires case-level metrics. Flatten into rows:

{
  "run_id": "...",
  "model": "...",
  "case_id": "...",
  "status": "...",
  "tokens": 0,
  "duration_ms": 0,
  "verbosity": 0.0,
  "accuracy": 0.0,
  "confidence": 0.0,
  "log_path": "_runs/<run>/logs/<case>.log"
}

Step 3: Load the canvas prompt template

Read templates/canvas/compare-prompt.md.tmpl. Do not modify it — forward it verbatim alongside the dataset.

Step 4: Emit the hand-off

Output exactly this JSON (stdout):

{
  "tool": "/canvas",
  "instructions": [ /* from the template */ ],
  "dataset": [ /* flattened rows */ ]
}

Then tell the host agent: "Route this payload to /canvas — do not render charts in text."

Step 5: Drill-down

When the user selects a point on the rendered chart, the log_path field maps to {evalsDir}/<log_path>. The /canvas tool opens that file on click.

What NOT to Do

• Do not render charts in text or markdown. /canvas owns layout.
• Do not aggregate or downsample. Emit every row; /canvas decides.
• Do not call askQuestion.