zoto-help-evals

name

zoto-help-evals

description

Project-tailored help for the Eval System plugin. Loads plugins/zoto-eval-system/README.md as the source of truth, inspects current project state (.zoto/eval-system/config.yml, manifest.yml, .env*, evals/_runs/, package.json scripts), and composes an answer that is grounded in the chosen README section, tailored to the host repo's actual configuration and artefacts, and cited with `start:end:path` code references back to the README. Pre-collected `help_context` arrives from the /z-eval-help command (command-owned askQuestion). The skill itself never calls askQuestion — missing context is returned as `needs_user_input`.

Help Evals

Canonical help for the Eval System plugin. The README is the source of truth; this skill grounds every answer in it, and tailors the answer to the host project so users see their actual evalsDir, framework, model, manifest targets, and run history — not generic defaults.

Configuration

No repo-level config required. The skill uses:

• plugins/zoto-eval-system/README.md (or the installed path at ~/.cursor/plugins/zoto-eval-system/README.md) as the documentation source of truth.
• The host repo's eval-system state files for tailoring (read-only): .zoto/eval-system/config.yml, .zoto/eval-system/manifest.yml, .zoto/eval-system/manifest.history.yml, .env.example, .env, package.json (the eval* scripts), and the latest evals/_runs/<run-id>/ if any.

When to Use

• /z-eval-help invoked (after the command resolved which section to answer for).
• The routing rule rules/zoto-eval-system.mdc detects help-intent in the user's message and hands off here before the agent answers from inferred knowledge.

Workflow

Step 1: Load the README

Read the README from the plugin path. Parse ## section headers. The list of sections is the menu the command uses; the chosen section is the answer's anchor — every claim in the response must trace back to it (or to a clearly-cited adjacent section).

Step 2: Section selection (command-pre-collected)

The /z-eval-help command runs askQuestion before invoking this skill when:

• No <topic> was given → numbered section picker.
• Multiple headers match <topic> → disambiguation picker.
• Follow-up navigation (related sections) → optional second prompt + resume.

The Task prompt (or skill input) MUST include help_context:

• selected_section: exact ## header title or slug to anchor the answer on. Must match a README section verbatim — e.g. Strategy bridge, Configuration, LLM backend (@cursor/sdk).
• follow_up: optional enum (related, quick_start, done) if the command already collected the next navigation step.
• user_question: optional free-form follow-up the user typed instead of picking an option.

If selected_section is missing and cannot be inferred safely, return needs_user_input with candidate sections — never call askQuestion.

Step 3: Load project signals (mandatory before composing the answer)

Read whichever of these are relevant to the chosen section. Skip files that don't exist; never fabricate values. When a signal is unread, the answer must not assert anything about it.

Section anchor	Signals to read
Overview, Quick start	.zoto/eval-system/config.yml (presence), .zoto/eval-system/manifest.yml (presence), package.json (scripts.eval*)
Configuration	.zoto/eval-system/config.yml (full), templates/schema/config.schema.json for field reference
Static backend	config.json → static.framework, host repo's evalsDir contents
LLM backend (@cursor/sdk)	config.json → llm.*, .env.example, .env (presence only — never read secret values), package.json devDeps for @cursor/sdk, dotenv
Interaction branch (unified LLM eval harness)	manifest.yml → per-target eval_files[] (one co-located <kind>/evals/<name>.json per target); presence of evals/llm/_shared/vitest-json-loader.ts and evals/llm/_shared/run-llm-suite.ts; cached analyser payloads under .zoto/eval-system/cache/analyser/ for requiresInteraction; latest report.yml per-case backend:
Advanced TS escape hatch (runner cases) and scenarios	README "Eval formats" / "Advanced TS escape hatch" / "Multi-primitive scenarios" sections; evals/llm/_shared/runner-params.ts contract; evals/scenarios/ directory listing; presence of evals/scenarios/_example-multi-primitive.test.ts
Updating evals	manifest.yml → discovery_config, targets[]; manifest.history.yml (last entry)
Result schema	latest evals/_runs/<run-id>/report.yml and per-backend static.yml / llm.yml if any
Run logs	evals/_runs/ directory listing; latest <run-id>/logs/
Comparing runs	last 2 entries under evals/_runs/
Judge & soft metrics	config.json → judgeModel, latest llm.yml
CI integration	.github/workflows/*.yml if present, host package.json scripts
Troubleshooting	whichever signals match the symptom in user_question

If a signal disagrees with the README (e.g. host uses vitest static framework but the README's pytest example is the only one), prefer the project signal for facts about the host repo and the README for facts about the plugin's contract, and call out the mismatch explicitly.

Step 4: Compose a tailored, cited answer

The answer paraphrases freely and is grounded in the README. Three rules are non-negotiable:

1. Anchor on the chosen section. Every claim about plugin behaviour must trace to lines in the README. Adjacent sections may be cited if they're directly relevant to the user's question.

2. Tailor with project signals. Replace generic placeholders with values read from the host repo. Examples:

   • "your evalsDir is evals/" — only after reading config.json.
   • "you currently have N targets in the manifest" — only after reading manifest.yml.
   • "your latest run is <run-id>" — only after listing evals/_runs/. When a value cannot be read (file absent), say so and link to the command that creates it (e.g. "no manifest.yml yet — run /z-eval-create").

3. Cite every quoted block with the code-reference syntax, using line numbers from the README:

   ...the exact lines from README...
   

   Inline references to other files (e.g. the user's config.json) use the same start:end:path syntax. Filenames in prose are wrapped in backticks. Never invent line numbers — re-read the file if unsure.

Answer skeleton

<one-paragraph orientation, in your own words, anchored on the chosen section>

<key points as a short list — paraphrased, each followed by an inline citation
  to the README lines that back the claim>

<project-tailored observations from Step 3, e.g.
  "Your config has `static.framework: vitest`, so the pytest section above
   doesn't apply — see the vitest backend stanza instead." >

<README quote block(s), each in the start:end:plugins/zoto-eval-system/README.md
  code-reference format, kept short and on-topic>

<next-step suggestion: which command to run, or which other section to read>

Step 5: Follow-up navigation

If follow_up is absent but the user clearly needs another section, return needs_user_input with section options so the command can askQuestion and resume. Do not chain to a second section silently.

Citation format — required

• README quote: code reference with line numbers. Paths are relative to repo root (plugins/zoto-eval-system/README.md). Do not add a language tag.

  ## Quick start
  ...
  

• Project file quote (e.g. user's config.json): same start:end:path syntax.

• Inline reference in prose: backticked file path, e.g. plugins/zoto-eval-system/README.md — used when no quoted block is needed.

Anti-patterns (will fail the skill's evals):

• Pasting the entire README section without paraphrase or project tailoring.
• Asserting project-specific facts ("your evalsDir is evals") without reading the file first.
• Paraphrasing the README without any citation back to it.
• Inventing line numbers.

What NOT to Do

• Do not answer from inferred knowledge. The README is the contract; project files are the host state.
• Do not rewrite the README. Edits go through plugin authors committing directly.
• Do not call askQuestion from this skill. Use needs_user_input so the command can prompt and resume.
• Do not read or echo secret values from .env — only check whether the file exists.
• Do not invent project values when a file is absent. Say "no manifest.yml yet" and point at the command that would create it.

Q&A quick reference

User question	Answer anchor
How do I write a TS eval?	Use a runner case in your co-located JSON eval (see README "Advanced TS escape hatch"), or add a multi-primitive scenario under evals/scenarios/.
Where is the RunnerParams contract defined?	evals/llm/_shared/runner-params.ts (TS source) and plugins/zoto-eval-system/templates/schema/runner-params.schema.json (JSON Schema mirror).