n8n-create-instance-ai-eval

# Create an Instance AI eval Each eval is **one JSON case**. Author workflow cases in `packages/@n8n/instance-ai/evaluations/data/workflows/`. Author standalone Agent cases in `packages/@n8n/instance-ai/evaluations/data/agents/` and follow the [`create-agent-builder-eval` skill](../create-agent-builder-eval/SKILL.md). The disk loader auto-discovers `*.json` in both directories. A LangTracer suite is the durable home. Cases validate against [`harness/schema.ts`](../../../packages/@n8n/instance-ai/evaluations/harness/schema.ts) (`.strict()` — unknown keys fail at load). The eval [README](../../../packages/@n8n/instance-ai/evaluations/README.md) is the exhaustive field reference; this skill is the opinionated *how*. > **Committing new case JSONs into the repo is no longer the recommended > approach.** Author the file locally (uncommitted), calibrate it against a real > build, then **push it to a lang-tracer suite** with `eval:langtracer-push` > (see [Push to a lang-tracer suite](#push-to-a-lang-tracer-suite)) — > the suite the driver chose (see > [Ask for the target suite first](#ask-for-the-target-suite-first)) — > `baseline` for the consolidated corpus n8n CI runs, or a dedicated > capability suite like `agents`. > The suite is the home for the case; the eval CLI reads it back via > `--source langtracer`. You still write the JSON file — it's just the input to > the push, not a committed artifact. > > **Seeded cases.** An `inline` seed pushes with the case — the case-write API > stores it verbatim, so the suite is its home like any other case. Only a > `seed.mode: "replay"` case is refused (listed under `skipped:`): it's > reconstructed from a LangSmith trace at run time, so it dies when that trace is > pruned and has no durable home. Don't commit a replay case either — derive a > synthetic case from it. See [`case-shapes.md`](case-shapes.md). ## Ask for the target suite first **Before you source, draft, or run anything, ask the driver which LangTracer suite receives every eval you create in this session.** Ask this in both autonomy levels, including autonomous mode. Never pick the suite yourself and never default silently. A case pushed to the wrong suite runs in the wrong CI lane and alerts the wrong team. Ask the suite question and the autonomy question together, in one message. Recommend `baseline`: - `baseline` (recommended): the general corpus. The Instance AI (INS) team monitors it, and it runs nightly. - Any other suite, such as `agents` or `node-gauntlet`: call `list_suites` and offer the live list. Do not use a hardcoded list, because suites are added and retired. Rules: - The driver's answer applies to all cases in the session. Do not ask again per case. - If the driver already named a suite in the request, state it back in one line and continue. Do not ask again. - Check that the suite kind matches the case. Push a case with a wrong build (`capability_gap`) only into a suite that allows that kind. If the suite does not allow it, tell the driver and ask again. - Put the chosen suite slug in the final decision log, as a link. ## Set the autonomy level first **Before you source, draft, or run anything, decide how hands-on the driver wants to be — and say it back.** This skill runs at one of two autonomy levels. If the request makes the level clear ("just author and calibrate it yourself" vs. "stop me at each step", or an explicit mode), adopt it, state it in one line, and note how to override (e.g. "say 'stop me at calibration' to add a checkpoint"). **If it's not clear, ask the driver one question** offering the two levels *before* doing any work. The skill has four natural decision **gates** — **selection** (which real failure to encode), **shape + expectations** (archetype, must-haves, scope trim), **calibration** (classify each red and resolve keep/loosen/drop), and **push** (kind + tier). The level decides what happens at each gate: | Level | Who decides when to stop | Behaviour | |---|---|---| | **autonomous** | agent | Runs all four gates start-to-finish; reports a **decision log** at the end for the driver to review — with the pushed case, its suite, and the source thread as **links** ([Share links, never bare ids](#share-links-never-bare-ids)); includes the observation id when the driver opted to record one; and includes a Linear ticket **proposal** for any kept capability-gap red ([Capability gap → propose a Linear ticket](#capability-gap--propose-a-linear-ticket)). | | **checkpoint** | driver, per gate | Stops at each gate with a compact **proposal + recommendation**; driver says "go" or redirects. At the **calibration** gate, hands the driver a link to the just-built thread on the live instance plus login credentials so they can review the real conversation and workflow themselves before confirming (below). | **Calibration is special-cased at both levels.** A calibration verdict that flips a case's *meaning* — a real capability-gap red vs. a harness-caused red, or any loosening that would let a known-bad build pass — is **surfaced explicitly** (interactively in checkpoint; in the decision log in autonomous), never silently committed. It's the one call where a quiet mistake corrupts the suite, so it never fully auto-commits. **Recording a source observation is optional, not a fifth gate.** After a real thread passes selection, offer to save why it was selected and what the developer observed with LangTracer's `create_observation` tool (see [`sourcing-cases.md`](sourcing-cases.md#optional-record-the-selection-as-an-observation)). If the driver declines, has not stated a preference in autonomous mode, or the write fails, continue with the eval. Never block drafting, calibration, or push on an observation. `create_observation` and `update_observation` write only to LangTracer; do not add a LangSmith feedback or sync step. **Checkpoint calibration — review the real thread on the instance.** Because the calibration verdict is trust-critical, in checkpoint mode you don't ask the driver to trust your reading of the run. You built the case against a live instance with `--keep-workflows` (step 4), so the thread and the workflow are still there — hand the driver a direct link and let them look: - **Thread:** `<base-url>/assistant/<threadId>` — the exact conversation the case ran (the run prints the `threadId`; the built workflow prints as `BUILT (<id>)` and opens at `<base-url>/workflow/<id>`). - **Login:** the email + password the eval signs in with (the owner you seeded on the instance — see [`running-evals.md`](running-evals.md); the default local seed is `nathan@n8n.io` / `PlaywrightTest123`). Present, per red: the assertion, whether it went green/red, your proposed classification (real capability gap / harness limitation / noise) and keep/loosen/drop, and the review link. The driver logs in, reads the thread and the workflow, and confirms or redirects before you write the verdict back into the case `description`. ## Share links, never bare ids Every lang-tracer entity has a shareable web page, but the CLI and the MCP hand you **numeric ids** — `eval:langtracer-push` prints `+ created <slug> (#621)`, `get_eval_run` returns a run number, `list_conversations` returns thread ids. An id is unclickable: the driver has to go find it. **Whenever you name a case, suite, thread, cluster, or run in anything a human reads** — a checkpoint proposal, the end-of-run decision log, a PR description, a Linear ticket, a Slack message — render it as a link, keeping the id in the label: ``` pushed as [#621](https://lang-tracer.n8n-maintenance.workers.dev/test-cases/621) ``` Build links off the **web base** (`LANGTRACER_URL`, in production `https://lang-tracer.n8n-maintenance.workers.dev`). Never off the API bases — `${LANGTRACER_URL}/api/v1` and `/api/mcp` are machine endpoints, and a link into either 404s for the driver or dumps JSON. | Entity | URL | Where the id comes from | |---|---|---| | Test case | `<base>/test-cases/<id>` | push output `(#<id>)`; `create_test_case` / `search_test_cases` | | Suite | `<base>/suites/<suiteId>` | push header `Suite "<slug>" (#<id>)`; `list_suites` | | Source conversation | `<base>/conversations/<threadId>` | `list_conversations` / `get_conversation` | | Cluster report | `<base>/clusters/<id>` | `list_cluster_runs` / `get_latest_cluster_run` | | Eval run (sweep) | `<base>/results?sweep=<runId>` | `list_eval_runs` / `get_eval_run` (`runId` *is* `sweeps.id`, the "run #N") | Two links that are **not** lang-tracer and don't take this base: the built thread (`<base-url>/assistant/<threadId>`) and workflow (`<base-url>/workflow/<id>`) live on the **n8n instance** the eval ran against. When both are relevant — reviewing a calibration red, writing a capability-gap ticket — give both, labelled, so nobody has to guess which host a link points at. ## Where the best cases come from The strongest cases encode a **real** failure, not an invented premise. Two connections help you find and verify one: **LangTracer** clusters real conversations into capability-gap themes (discover what actually fails, at scale), and **LangSmith** holds the raw traces (verify exactly what happened in a run). LangTracer is the discovery layer; the durable artifact is almost always a synthetic case you author from what you learn (use `seed.mode: "replay"` only per [`case-shapes.md`](case-shapes.md)). See [`sourcing-cases.md`](sourcing-cases.md) for connecting the MCPs and the discover → verify → encode workflow. ## Pick the case shape first The corpus is five archetypes. Decide which you're writing before you draft — it determines the fields, the grading, and how you validate. They compose (a seeded case can still assert outcome), but the primary shape drives the work. | Archetype | Question it answers | Primary fields | |---|---|---| | **Build** (default) | Does the workflow the agent builds actually *work*? | `outcomeExpectations` + `executionScenarios` | | **Behaviour / process** | Does the agent *converse* correctly (ask the right clarifying question, not re-ask, honour a correction, respect plan approval)? | `processExpectations` + multi-turn director script; often **build-only** | | **Credential** | Does the build behave correctly given a specific credential view? | `credentials[]` | | **Seeded** | Start mid-thread, with prior work already in place, and drive the turn under test | `seed` (authored `mode: "inline"`; `"replay"` for a local check) | | **Context** | Does the agent still work once the conversation is long — does it reuse what it already read, and keep what matters after observational memory compacts the history? | `processExpectations` + a long `seed`; `requiresMemoryCompaction` for the compaction ones | **Build** is documented in full below. The other four, the director-script vocabulary, and the seeding modes are in [`case-shapes.md`](case-shapes.md). The judge also sees **token ground truth** for workflow-build cases: per-turn input and output tokens inline in each transcript turn header, a build-wide total, a cache read/write split, and the opening step's cost. So an expectation may reference cost or consumption directly ("does not re-read the same node's schema in turn 2"). The numbers cover the orchestrator's own LLM steps only. A delegated Agent build runs in a sub-agent whose steps are not in the snapshot, so do not write cost expectations on an Agent case. Those numbers come from run-debug snapshots, so they need `N8N_INSTANCE_AI_RUN_DEBUG_ENABLED=true` on the instance under test; without it the judge reads `(no run debug captured)` and cost expectations are ungradeable. The judge also sees the thread's **observation rows** — what observational memory kept after it compacted — so a memory case can grade the summary itself. That block is a separate REST read and needs no flag. ## Core principle (all shapes) **Write expectations from intent, then calibrate against a real build.** Decide up front what makes *any* correct solution correct — the must-haves implied by what the user actually said — then build the workflow once for real to calibrate granularity: loosen what's over-specified, confirm the must-haves are achievable, and catch requirements the agent legitimately satisfies a different way. Don't transcribe one observed build into assertions — that overfits the eval into "did the agent reproduce that run" instead of "did it solve the problem." **Keep the conversation in the user's voice.** State the goal and real constraints the way a user would — don't name node types, wire up the structure, or restate your `outcomeExpectations` in the prompt. If the conversation spells out the build, the case only tests whether the agent can follow instructions and the expectations become tautological; the gap between what the user asks for and how a correct workflow realizes it is the capability under test. Even when the anchor *is* honoring a user's stated technical preference, phrase it as their need + constraint ("I need field X and the built-in node doesn't expose it, so pull it straight from the API") — not as an implementation spec ("use an HTTP Request node"). **Write the conversation in English** unless the user asked otherwise (or the case exists specifically to test non-English handling). Sourced real threads are frequently non-English — translate the intent into English when you rewrite the prompt in the user's voice; the failure mode is the anchor, not the original language. **Trim to the smallest multi-turn conversation that reproduces the issue.** Real sourced threads are long (dozens of turns of setup, debugging, and tangents) — do **not** transcribe them. Distill to the fewest turns that still drive the build or behaviour under test. Every retained turn must earn its place: a turn stays only if it is *load-bearing* — a value the agent must ask for (withheld until asked, via a director note), a correction/push-back the case exists to test, or a plan approval that gates the build. If removing a turn doesn't change what's tested, remove it. **Collapse to a single turn** whenever the whole request can be stated at once without a load-bearing exchange; keep it multi-turn *only* for those exchanges, and keep each director script in one turn (don't fabricate assistant "done" turns to sequence steps — see [`case-shapes.md`](case-shapes.md)). A minimal conversation isolates the capability; a transcribed one buries it in noise and tests instruction-following. **Size the build, not just the assertions.** Real sourced prompts are often kitchen-sink ("production-ready, runs forever, 3 feed posts *and* 8 stories a day", "generate 50 articles daily") and reliably blow the ~900s build budget (see "Known harness limitations"). A **faithful trim is a legitimate authoring move**: reduce batch sizes, drop one of several parallel pipelines, or merge adjacent AI steps so the case builds within budget — then note the reduction in the case `description` ("the original request also asked for an 8-stories/day pipeline; scoped to feed posts so it builds in budget"). Keep the capability under test; cut the combinatorial bulk. A case that never builds tests nothing. ## Workflow These steps map to the four gates from [Set the autonomy level first](#set-the-autonomy-level-first): sourcing (before step 1) is the **selection** gate, including the optional offer to record a source observation; steps 1–2 are the **shape + expectations** gate; steps 5–6 are the **calibration** gate; steps 7–8 are the **push** gate. In *autonomous* mode you flow through all of them and summarize in a decision log; in *checkpoint* mode you pause at each with a proposal, and at calibration you hand the driver the thread link + login to review the real build (see [Set the autonomy level first](#set-the-autonomy-level-first)). Calibration (step 6) always surfaces meaning-flipping verdicts explicitly regardless of level. 1. **State the must-haves first.** From the conversation alone, list what every correct workflow must do (trigger type, essential operations, gating condition) — those become draft `outcomeExpectations`. Required fields: `conversation` (≥1 turn, first `user`), `complexity`, `tags`, and **at least one** of `executionScenarios` / `processExpectations` / `outcomeExpectations`. 2. **Draft the case** from the template below; validate it loads (see "Validate"). 3. **Smoke-test the *environment* with one case before any batch.** Run a single case end-to-end first. This validates auth / model / `--base-url` / the built dist for ~1/Nth the cost — distinct from validating a *case*. If that one case crashes at execution (especially with an identical error you'd expect to hit every case), fix the environment before running the batch (see "A red is signal" → environment check). Running 15 cases only to discover a stale-dist crash on all of them wastes a full run. 4. **Build it once** against a running instance (see [`running-evals.md`](running-evals.md)) with `--keep-workflows` so the built workflow stays for inspection. 5. **Inspect** — read the built workflow (the run prints `BUILT (<id>)`; fetch via `GET /rest/workflows/<id>`) and the HTML report's transcript to see what the agent actually did. 6. **Calibrate — sharpen assertions; never dull them to force a green.** Fix assertions that are genuinely mis-sized: relax one that pins a choice the conversation left open (so a valid *alternative* build wrongly fails), tighten one a wrong build would slip past, and phrase `executionScenarios` to match how the workflow runs on mocked data. But when a scenario goes red because the build has a real gap, or because the harness can't exercise it, **that red is the result — keep it and surface why** (see "A red is signal", below). Never delete a scenario, weaken an assertion, or drop to build-only just to make the run green. And when the case comes back **green**, that is a result to earn, not to accept: confirm the precondition actually fired, then re-derive it from the raw thread before calling the case a regression guard (see [First reproduce, then reclassify](#first-reproduce-then-reclassify)). 7. **Push to the suite — do NOT commit the JSON.** Once calibrated, push the case into its curated lang-tracer suite with `eval:langtracer-push` (see [Push to a lang-tracer suite](#push-to-a-lang-tracer-suite)); the suite is the case's home, not the repo. Leave the `data/{workflows,agents}/*.json` file uncommitted (or delete it once it's in the suite). Committing new case JSONs into the repo is no longer the approach. (An `inline` seed pushes with the case; only a `replay` case is refused — it's a local throwaway; see [`case-shapes.md`](case-shapes.md).) For a sourced case, finish by **linking it to its source thread/finding** over the MCP — see