Scaffold a new LLM evaluation suite package with Playwright config, evaluate fixture, and package files. Use when creating a new eval suite, adding an evals package for a plugin, or setting up the boilerplate for offline LLM evaluations.
Scaffold a new LLM evaluation suite package with Playwright config, evaluate fixture, and package files. Use when creating a new eval suite, adding an evals package for a plugin, or setting up the boilerplate for offline LLM evaluations.
Create an Eval Suite
Overview
Eval suites live in dedicated kbn-evals-suite-<name> packages. Each suite is a self-contained Playwright project that uses the evaluate fixture from @kbn/evals to run LLM experiments with datasets, tasks, and evaluators.
Inputs to Collect
Suite name (kebab-case, e.g. my-feature)
Parent directory under x-pack/ (e.g. x-pack/platform/packages/shared/ai-infra/ or x-pack/solutions/security/test/)
Owner GitHub team handle (e.g. @elastic/search-ml-ux)
Group (platform, security, observability, search)
Visibility (shared or private)
Whether custom fixtures are needed (chat client, esArchiver, supertest, etc.)
Do NOT Use node scripts/scout.js generate
Eval suites are not standard Scout test configs. The Scout generator creates directories that are picked up by Scout's CI discovery glob -- this will break because evals configs use (not ) and contain non-JS files (like prompt files) that Playwright cannot parse.
test/scout/
createPlaywrightEvalsConfig
createPlaywrightConfig
.text
The Scout team has explicitly asked that eval configs live outsidetest/scout/ directories. All eval suites place their playwright.config.ts in the package root.
Directory Layout
kbn-evals-suite-<name>/
├── evals/
│ └── <name>.spec.ts # evaluation spec(s)
├── src/
│ └── evaluate.ts # re-export or extend the base evaluate fixture
├── playwright.config.ts # MUST be in package root, NOT under test/scout/
├── package.json
├── kibana.jsonc
└── tsconfig.json
Needs HTTP chat client and external Phoenix executor
security-solution-evals
Extended with chatClient, esArchiver, supertest, quickApiClient
Domain-heavy setup: loads ES archives, uses generated API client
Suite Registration
Add an entry to .buildkite/pipelines/evals/evals.suites.json:
{"id":"<name>","name":"<Human Readable Name>","configPath":"<repo-relative path to playwright.config.ts>","tags":["<group>","<name>"],"ciLabels":["evals:<name>"]}
Registration is optional for local dev (suites are auto-discovered from createPlaywrightEvalsConfig imports), but required for CI labeling and node scripts/evals list.
Post-Scaffold Steps
Run yarn kbn bootstrap to register the new package.
Verify the suite appears: node scripts/evals list.
Create your first spec file under evals/ (see the evals-write-spec skill).
Run locally: node scripts/evals start --model <connector-id> --judge <connector-id>.
Common Mistakes
Placing configs under test/scout/ -- Scout's CI discovery will find them and crash. Keep playwright.config.ts in the package root.
Using node scripts/scout.js generate -- this creates Scout test scaffolds, not eval suites. Scaffold manually using the templates above.
Setting type to anything other than "functional-tests" in kibana.jsonc.
Forgetting @kbn/evals in kbn_references -- causes TS resolution failures.
Using Path.join instead of Path.resolve for testDir -- Playwright needs an absolute path.
Creating evals/ specs that import from @kbn/evals but the suite's src/evaluate.ts re-exports a different fixture -- always import evaluate from the suite's own src/evaluate when extending.
Forgetting to run yarn kbn bootstrap after creating the package.
