Menu
Invocation reference for all eval-system CLI commands. Use this skill when you need to run eval scripts — never reference script paths directly; always invoke via the package.json aliases documented here.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Baseado na classificação ocupacional SOC
Guided workflow for launching and interpreting the cursor-top TUI. Covers process discovery, hierarchy expansion, log tail interpretation, and recommended flags for different scenarios. Use when the user wants to monitor every Cursor agent on their machine in real time, or to capture a snapshot for a report.
Writes .zoto/eval-system/config.yml from field values pre-collected by /z-eval-configure (command-owned askQuestion). Refuses bundled update.preserveUserAuthoredCases:false or update.writeMetaMarker:false (including nested payload mirrors) before any manifest read or config write. Validates against templates/schema/config.schema.json. Diffs the chosen config against the manifest snapshot (discovery_config.static.framework) to produce a cleanup_plan validated against templates/schema/cleanup-plan.schema.json, and stamps _meta.primitive_analysis.invalidate=true on every cached primitive analysis when the static framework changes. The unified LLM eval harness ships one JSON-first co-located eval per target, with no per-repo selector axis — only the static framework matters here. Never calls askQuestion — returns needs_user_input if validation cannot proceed without user clarification.
Scaffolds static pytest and LLM (@cursor/sdk) eval backends into a host repository, generates eval cases with _meta.generated markers for approved skills/commands/agents/hooks, merges eval scripts into package.json, and writes the persistent manifest plus append-only history. Expects the invoking command to pre-collect approved targets — does not call askQuestion.
Executes eval runs via the host repo's package.json scripts (pnpm run eval, eval:full, eval:llm, eval:judge). Forwards --model to the LLM runner via both a CLI flag and the ZOTO_EVAL_MODEL environment variable. After each run, calls eval:update --check and appends the drift status to the run report as a warn-only line. Does not call askQuestion — credential gaps use needs_user_input for the command.
Adversarial judge for eval runs. Identifies weak graders, under-covered assertions, verbosity spikes, and accuracy/confidence regressions. Produces an enriched llm.yml with soft-metric annotations. Does not call askQuestion — handoff to /z-eval-update is expressed as needs_user_input for the command to confirm via askQuestion and resume. Uses the configured judgeModel (default claude-opus-4-8[]).
Diff-aware eval updater. Detects drift between covered targets and generated eval cases for skills and central plugin eval files (commands/agents/hooks), classifies changes as critical or non-critical, and preserves user-authored cases verbatim. Apply-mode regeneration re-invokes the LLM analyser per drifted primitive and dispatches per-framework static stamping plus the single unified LLM regeneration (`regenerateLlm`) (CI defaults to cached analyser payloads unless `--with-analyser`). The `_meta.generated === true` (case) and `// _meta.generated\: true` (file) contracts are enforced at runtime AND compile time. Supports targeted mode, rediscovery mode, `--no-analyser` / CI cache reuse, and CI `--check` (exit 2 on critical drift).
| name | zoto-eval-tooling |
| description | Invocation reference for all eval-system CLI commands. Use this skill when you need to run eval scripts — never reference script paths directly; always invoke via the package.json aliases documented here. |
This skill is the single source of truth for how agents invoke eval-system CLI commands.
Lean hosts (default): nested .zoto/eval-system/package.json exposes pnpm run eval and pnpm run eval:full (root may delegate via pnpm -C .zoto/eval-system run eval). Every other command is invoked via the bridge from the repo root:
tsx .zoto/eval-system/scripts/eval-bridge.ts <script-base> [-- <args>]
Examples: tsx .zoto/eval-system/scripts/eval-bridge.ts eval-discover, tsx .zoto/eval-system/scripts/eval-bridge.ts engine/update -- --check, tsx .zoto/eval-system/scripts/eval-bridge.ts eval-stamp -- command:foo.
Ejected hosts: root aliases delegate to pnpm -C .zoto/eval-system <script>; the nested package carries the full eval:* set.
Agents MUST NOT reference underlying TypeScript file paths in the plugin install — always go through .zoto/eval-system/scripts/eval-bridge.ts (lean) or root pnpm run eval:* (ejected).
pnpm run eval:discoverDiscover eval targets in the host repository.
| Detail | Value |
|---|---|
| Purpose | Walk configured skillsRoots and discoveryTargets, emit a manifest-shaped YAML payload |
| Args | -- --resolve <file> — resolve a single file to its target ID(s) |
| Output | YAML on stdout (manifest shape) |
| Exit codes | 0 = success, 1 = missing config |
| Config fields used | skillsRoots, discoveryTargets, evalsDir, ignore |
pnpm run eval:analyseRun the LLM-driven primitive analyser on one or more targets.
| Detail | Value |
|---|---|
| Purpose | Produce an AnalyserPayload (JSON) per primitive via @cursor/sdk |
| Args | -- <target-id> [--target <glob>] [--pretty] [--dry-run] |
| Output | JSON on stdout per target; cache written to .zoto/eval-system/cache/analyser/ |
| Exit codes | 0 = success, 1 = error, 2 = budget exhausted |
| Config fields used | llm.model.id, analyser.concurrency, analyser.maxCallsPerInvocation, ignore |
| Schema | Output validated against templates/schema/analyser-payload.schema.json |
pnpm run eval:stampStamp (generate) per-primitive eval test files from an analyser payload.
| Detail | Value |
|---|---|
| Purpose | Emit deterministic test files for pytest / vitest / jest backends (and, for one allowlisted skill, merge analyser rows into that skill’s evals/evals.json) |
| Args | -- <target-id> [--dry-run] [--baseline-only] |
| Output | Files written under {evalsDir}/ for command/agent/hook targets; skill:zoto-eval-tooling stamps plugins/zoto-eval-system/skills/zoto-eval-tooling/evals/evals.json from the analyser cache — stdout reports the path written |
| Exit codes | 0 = success, 1 = error (missing payload, bad target), 2 = non-allowlisted skill:* target |
| Config fields used | static.framework, evalsDir |
| Note | skill:zoto-eval-tooling is allowlisted: run pnpm run eval:analyse -- skill:zoto-eval-tooling first (cache under .zoto/eval-system/cache/analyser/), then pnpm run eval:stamp -- skill:zoto-eval-tooling. Every other skill:* refuses stamp — use templates/skill-evals/evals.json.tmpl. The allowlist (CENTRAL_STAMP_SKILL_ALLOWLIST) lives in eval-stamp.ts: lean mode resolves it from the plugin via eval-bridge.ts (plugins/zoto-eval-system/scripts/eval-stamp.ts); ejected mode uses the vendored copy at .zoto/eval-system/scripts/eval-stamp.ts. |
pnpm run eval:cleanup-staleSafe workflow before deleting generated eval assets after a static framework change: default mode only plans; destructive --apply is gated. The unified LLM eval harness ships a JSON-first backend shape (one co-located <kind>/evals/<name>.json per target), so cleanup only sweeps stale static-framework fingerprints / tests and orphaned co-located <kind>/evals/<name>.json files (and orphaned generated rows in central evals.json).
| Detail | Value |
|---|---|
| Purpose | Compute a cleanup_plan (manifest snapshot vs active .zoto/eval-system/config.yml) listing stale fingerprints, tests, and eval JSON rows eligible for removal or surgical rewrite |
| Default (no mode flag) | Same as --dry-run: prints the full cleanup_plan JSON on stdout (pretty-printed). The engine always validates the emitted document before printing. Stderr includes the authenticated cleanup session lines (runId, plan_hash, TTL) when a lockfile is written under evals/_runs/ (suppress in tests with --no-lockfile). |
-- --dry-run | Explicit dry-run; same stdout/stderr contract as the default. Mutually exclusive with --apply and --check. |
-- --check | CI / pre-merge drift gate: recomputes and validates the plan, but does not print the full plan on stdout. Exit 0 when cleanup_plan.totals.files === 0 (working tree matches the new config — no stale assets). Exit 2 when totals.files > 0 (drift); stderr summarizes how many stale files / groups were found. Mutually exclusive with --dry-run and --apply. |
-- --apply | Performs deletions / surgical rewrites from the computed plan. Guarded: unless --force, you must pass at least one of --session <runId> or --token <plan_hash> from the prior --dry-run stderr (lockfile under evals/_runs/.cleanup-token-<runId>.json). The engine verifies TTL (1 hour) and that the live plan_hash still matches the lockfile. --force skips that gate (non-interactive / migration only; logs a warning). Stdout on success is a short JSON summary (applied, counts, warnings), not the full cleanup_plan. |
-- --force | Only meaningful with --apply; bypasses session/token gate. |
| Output | Full cleanup_plan JSON on stdout in default / --dry-run only; --check uses exit code + stderr; --apply prints apply summary JSON on stdout |
| Exit codes | 0 — success (dry-run printed, check found no drift, or apply completed). 1 — invalid mode mix, missing session/token on apply, expired/missing lockfile, plan hash mismatch vs lockfile, refused paths, or other runtime errors. 2 — --check detected stale files (totals.files > 0). 3 — emitted plan failed AJV validation against the cleanup-plan schema (engine/schema drift). |
| Config fields used | static.framework, ignore (removed-target group) |
| Schema | Printed and internally validated against plugins/zoto-eval-system/templates/schema/cleanup-plan.schema.json (same path as templates/schema/cleanup-plan.schema.json relative to the eval-system plugin). This is the schema /z-eval-configure and the configurer skill use for cleanup_plan. |
pnpm run eval:updateDiff-aware refresh of generated eval cases (re-discovers, detects drift, optionally applies).
| Detail | Value |
|---|---|
| Purpose | Detect drift in AI primitives and update generated eval cases with user confirmation |
| Args | -- [--apply] [--check] [--target <glob>] [--no-analyser] [--with-analyser] (CI defaults to --no-analyser semantics unless --with-analyser) |
| Output | Drift report on stdout; manifest updated on apply |
| Exit codes | 0 = success (clean check), 2 = critical drift (check), 5 = update.failOnNoAnalyserInCI forbids CI cached-analysis mode |
| Config fields used | update.*, discoveryTargets, skillsRoots, ignore |
pnpm run eval / pnpm run eval:fullOrchestrate a full eval run (static + optional LLM backends).
| Detail | Value |
|---|---|
| Purpose | Run configured backends, write static.yml + llm.yml + report.yml per run |
| Args | -- [--full] [--llm-only] [--model <id>] |
| Output | Run folder at {evalsDir}/_runs/<ts>/ |
| Exit codes | 0 = success |
| Config fields used | evalsDir, static.framework, llm.model.id, runs.retention |
pnpm run eval:gcPrune old run folders past the configured retention limit.
| Detail | Value |
|---|---|
| Purpose | List (dry-run) or delete past-retention run directories |
| Args | -- [--dry-run] [--apply] [--retention <N>] |
| Output | JSON plan on stdout |
| Exit codes | 0 = success |
| Config fields used | evalsDir, runs.retention |
pnpm run eval:stamp-host-layoutOpt-in eject — copy self-contained runtime into .zoto/eval-system/.
| Detail | Value |
|---|---|
| Purpose | Vendor src/, templates/, engine/, scripts/ under .zoto/eval-system/, write nested package.json, stamp eval primitives to .cursor/*/eval-sys/, set hostLayout: ejected |
| When | Offline/air-gapped hosts, heavy engine customisation, CI without plugin access — not part of /z-eval-create |
| Args | -- [--dry-run] |
| Output | Summary on stdout listing copied dirs and config patch |
| Exit codes | 0 = success, 1 = error |
| Config fields used | Patches hostLayout to ejected |
| Note | In lean mode, this alias invokes plugins/zoto-eval-system/scripts/stamp-host-layout.ts directly (not via eval-bridge.ts). After eject, other eval:* aliases run scripts inside .zoto/eval-system/. |
pnpm run eval:un-ejectReverse eject — restore lean plugin-dependent layout.
| Detail | Value |
|---|---|
| Purpose | Remove vendored runtime, strip ejected primitives from .cursor/, set hostLayout: plugin, re-merge root eval aliases |
| When | Returning to lean after a prior eject; external repos migrating off full-stamp create |
| Args | -- [--dry-run] [--force] |
| Preserves | config.yml, manifest.yml, eval cases, {evalsDir}/_runs/ |
| Exit codes | 0 = success, 1 = error or user cancelled |
| Note | Requires resolvable plugin (or ZOTO_EVAL_PLUGIN_ROOT) before subsequent lean-mode eval runs. Run pnpm install after un-eject. |
| Mode | hostLayout | Script resolution |
|---|---|---|
| Lean (default) | plugin | pnpm -C .zoto/eval-system run eval / eval:full (or root delegates); other commands → tsx .zoto/eval-system/scripts/eval-bridge.ts <script-base> |
| Ejected (opt-in) | ejected | Root aliases → .zoto/eval-system/scripts/<name>.ts (nested package) |
Agents MUST use pnpm run eval:* aliases — never hard-code paths that assume one mode.
pnpm -C .zoto/eval-system run eval / eval:full (or root pnpm run eval when delegated) for orchestrated runs; use tsx .zoto/eval-system/scripts/eval-bridge.ts <script-base> [-- args] for everything else — never pnpm exec tsx paths under the plugin install.pnpm run eval:* aliases (they delegate to .zoto/eval-system/).-- when the script needs them: tsx .zoto/eval-system/scripts/eval-bridge.ts eval-analyse -- command:foo.loadEvalConfig). You do NOT need to pass --config.hostLayout: plugin) resolves scripts via .zoto/eval-system/scripts/eval-bridge.ts; ejected (hostLayout: ejected) runs vendored scripts under .zoto/eval-system/. Eject/un-eject are CLI-only (tsx .zoto/eval-system/scripts/eval-bridge.ts stamp-host-layout / eval-un-eject in lean mode)..zoto-eval-system/ directory, it is migrated transparently on first run.{ "error": "...", "code": "..." }).