| name | writing-evals |
| description | Write and iterate on tool-selection evals for the Vantage MCP server — evalite setup, prompt matrix, distractors, failure diagnosis, and db workflow. Use when adding or updating evals under `evals/` or when an eval run fails after changing a tool description or zod schema. |
Writing evals
Unit tests prove a tool wires up and the API call shape is right. Evals prove the description + zod schema are good enough that a model can find and call the tool from a natural-language prompt. Tool authoring conventions (description style, zod .describe() strings) live in .agents/skills/writing-mcp-tools/SKILL.md; this skill covers the eval harness and how to iterate when rows fail.
Stack and commands
evalite v1 + Vercel AI SDK v6 + @ai-sdk/anthropic + @ai-sdk/openai.
| Command | Purpose |
|---|
npm run eval -- ./evals/<path>.eval.ts | Run one eval file (normal workflow) |
npm run eval | Run every eval file — only when refreshing the whole baseline |
npm run eval:dev | Watch mode + UI on localhost:3006 |
npm run eval:export | Static HTML from current evalite.db (no re-execution) |
Evalite auto-loads .env; set ANTHROPIC_API_KEY and OPENAI_API_KEY before the first run. Model outputs are cached by default, so iterating on a description only repays LLM cost for changed cells.
evalite.config.ts sets forceRerunTriggers: ["src/tools/**/*.ts", "evals/_lib/**/*.ts"] — editing a tool or shared eval lib invalidates cache for affected runs. That is intentional when using eval:dev.
Persistence and what to run
Results are stored in evals/evalite.db (SQLite, committed to the repo as a shared baseline). The workflow is explicit, not automatic — evalite does not detect which tools have changed and re-run their evals for you. Every cell carries a created_at timestamp; treat it as the audit trail for "when did this eval last run vs. when did the tool last change."
- Adding a tool: run only the new eval file:
npm run eval -- ./evals/<...>/<your-tool>.eval.ts. Existing tools' rows stay untouched. Commit the updated evalite.db alongside the tool change.
- Editing an existing tool: re-run that tool's eval file the same way. The new rows append; older rows remain for history.
- Avoid
npm run eval with no path during normal development — it re-executes every eval file and appends a fresh row set for each. Use it only when intentionally refreshing the whole baseline.
- WAL sidecars (
evalite.db-wal, -shm, -journal) are ignored in .gitignore. Only the .db file is tracked. Don't try to commit the sidecars — they're per-process state and will corrupt the db for the next reader.
- Merge conflicts on the
.db are binary and can't be auto-resolved. If two branches both updated it, the simplest fix is to take one side, re-run the affected eval files on the merged branch, and commit the result.
- Browsing results: a live UI of the committed db is published to GitHub Pages at https://vantage-sh.github.io/vantage-mcp-server/ on every push to
main that touches evals/evalite.db. Locally, npm run eval:export produces the same static bundle under ./evalite-export/ without re-executing anything.
Layout
evals/
_lib/
models.ts # haiku 4.5, sonnet 4.6, gpt-5-mini, gpt-5 (wrapped with wrapAISDKModel)
distractors.ts # 5-tool pool for "mixed" mode + pickTools(target, mode)
variants.ts # cartesian product of {model} × {isolated, mixed} = 8 variants
buildAiSdkTools.ts # reads from the live registerTool registry → AI SDK tool() defs
runToolSelection.ts # { prompt, model, toolNames } → { toolCalls, text }
<tool-name>.eval.ts # for top-level tools
<resource>/<tool>.eval.ts # for nested tools
evalite.config.ts
Top-level evals/ keeps *.eval.ts out of Vitest's path. The adapter imports src/tools once and reads tools out of the live registerTool registry — adding a new tool to the codebase makes it available to evals automatically; you just need to write its eval file.
The matrix
Every tool's eval file contains two suites — one for direct phrasing, one for inferred phrasing — and each suite runs against 8 variants (4 models × 2 loading modes). That's a 4-quadrant matrix per prompt:
| Isolated (only the target tool loaded) | Mixed (target + 4 distractors) |
|---|
| Direct | Can the model pick the tool when the user names the concept and nothing competes? | Same wording, but is the description distinct enough that 4 unrelated neighbours don't pull it astray? |
| Inferred | Does the description cover the indirect phrasing well enough to fire at all? | Both pressures combined — the realistic deployment case. |
Each cell is what diagnoses a failure (see "Reading failures" below). Caps out at 64 cells per tool with 4 prompts per suite, which is what evals/get-myself.eval.ts runs in ~10s on a warm cache.
File template
evals/get-myself.eval.ts is the canonical reference. The shape per tool:
import { evalite } from "evalite";
import { toolCallAccuracy } from "evalite/scorers";
import { pickTools } from "./_lib/distractors";
import { runToolSelection } from "./_lib/runToolSelection";
import { TOOL_EVAL_VARIANTS } from "./_lib/variants";
const TARGET = "<tool-name>";
const directPrompts = [
{ input: "<prompt naming the concept directly>", expected: [{ toolName: TARGET, input: {} }] },
];
const inferredPrompts = [
{ input: "<prompt where intent has to be inferred>", expected: [{ toolName: TARGET, input: {} }] },
];
evalite.each(TOOL_EVAL_VARIANTS)(`${TARGET} · direct phrasing`, {
data: () => directPrompts,
task: (prompt, variant) =>
runToolSelection({
prompt,
model: variant.model,
toolNames: pickTools(TARGET, variant.mode),
}),
scorers: [
({ output, expected }) =>
toolCallAccuracy({
actualCalls: output.toolCalls,
expectedCalls: expected ?? [],
mode: "flexible",
}),
],
});
Use toolCallAccuracy in flexible mode for selection — order doesn't matter, only that the right tool was called with the right args. Reserve exact mode for multi-step workflow evals where call sequence matters.
Writing prompts
- Direct prompts name the concept the tool covers — "list my budgets", "show recommendation views", "delete cost report crt_xyz". They test that the description's first sentence carries the load.
- Inferred prompts describe the user's goal, not the tool — "I want to make sure we don't blow past $50k this quarter" →
create-budget. They test description coverage and any disambiguating context.
- Write prompts a Vantage MCP user would actually send. Generic phrasings with no product context (e.g.
"Who am I?") put unfair pressure on the description — models may read them as general knowledge questions, not Vantage account queries. Drop or rephrase prompts like that rather than padding the tool description to catch them.
- 3–6 prompts per suite is plenty. More cells = more API spend on every model rev, not more signal.
Distractors
_lib/distractors.ts defines a fixed 5-tool pool — pickTools(target, "mixed") returns the target + 4 others (the target is filtered out of the pool first, so you always get exactly 4 distractors). The defaults are deliberately broad — they catch "is the description distinct from completely unrelated tools," not "is the description distinct from a sibling tool." If your tool has close neighbours (e.g. list-budgets vs. list-folders vs. list-cost-reports), the default pool won't apply enough pressure. Two options:
- Pick siblings as the "distractors" inline in the eval — call
runToolSelection with toolNames: [TARGET, "list-folders", "list-cost-reports", ...] instead of going through pickTools.
- Add a tool-specific distractor list inside the eval file and use it for the "mixed" mode.
Either way, document the choice in a comment at the top of the eval file so the next person knows what the mixed mode is actually testing.
Reading failures
The matrix tells you what to fix:
| Failure pattern | Most likely cause | Fix |
|---|
| Direct + isolated fails | First sentence of description doesn't name the concept, or zod arg name buries it. | Edit description; rename/re-describe the arg. |
| Direct + mixed fails but isolated passes | A distractor's description is winning the comparison. | Add a one-line "Do not use for X" to either tool — usually the distractor. (See create-cost-alert.ts for the pattern.) |
| Inferred + isolated fails | Description doesn't cover the indirect phrasing. The arg names alone weren't a hint. | Add one sentence connecting the goal to the tool (cap: one sentence). |
| Inferred + mixed fails but isolated passes | Description covers the concept but a sibling tool covers it too well. | Disambiguate (same pattern as the second row). |
| Only the smallest model fails one prompt | Often a prompt-fairness issue, not a description issue. | Drop or rephrase the prompt. Don't grow the description to win a single weak-model row. |
| Wrong args (right tool) | A zod field is missing a useful .describe(), or a required field looks optional. | Tighten .describe() strings; add .default() if the value is genuinely defaultable. |
Iteration order when an eval fails:
- Zod schema first — better
.describe() on the args, tighter constraints (min(1), enum(...)), or a default(...) where the model shouldn't have to guess.
- Tool description second — add one sentence if needed; do not write paragraphs to win a single failure.
- Prompt third — if the prompt isn't realistic, fix the prompt rather than the description.
- Accept the floor — if only the smallest model fails on a fair prompt after the above, that's a model floor, not a tool bug.
The rule is: don't write to the eval. The eval validates the description and schema; making the description longer just to game one row defeats the point. If you're tempted, double-check the prompt is one a user would actually send.
Checklist
