Skip to main content
evals-write-spec Write LLM evaluation spec files with datasets, tasks, and evaluators using the @kbn/evals Playwright fixture. Use when authoring new eval specs, adding datasets or evaluators, or debugging evaluation test failures.
الانتقال إلى التثبيت سوق المهارات اكتشف واستكشف مهارات الذكاء الاصطناعي التي بناها المجتمع.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
عرض تفاصيل Prompt يتجاوز الأمر المباشر Prompt المخصّص للمراجعة. افحص المصدر قبل تشغيله.
npx skills add https://github.com/elastic/kibana --skill evals-write-specيبقى الأمر في سطر واحد. مرّر أفقيًا لمراجعته كاملًا قبل النسخ.
تفضّل نسخة محلية؟ نزّل الملفات المتاحة حاليًا لدى SkillsMP.
تحميل Zip جاري التحميل... المزيد من هذا المستودع Investigate Scout and FTR flaky test failures in Kibana. Use when triaging a failed-test issue, a Buildkite-reported failure, a test path that has been failing intermittently, or any time the user asks to look at a flaky test, deflake a test, or stabilize a test.
Scaffold a new LLM evaluation suite package with Playwright config, evaluate fixture, and package files. Use when creating a new eval suite, adding an evals package for a plugin, or setting up the boilerplate for offline LLM evaluations.
cypress-to-scout-migration Migrate Kibana Cypress E2E tests (.cy.ts) to Scout (Playwright). Applies to any Kibana plugin or solution. Includes triage gates (duplicate detection, layer analysis, value assessment), Cypress-to-Scout pattern mapping, data cleanup audit, and PR workflow. Use when: (1) migrating a Cypress test to Scout, (2) converting .cy.ts to .spec.ts, (3) planning a Cypress-to-Scout migration batch, (4) rewriting Cypress screens/tasks as Scout page objects, (5) asked "how do I move this Cypress test to Scout/Playwright", (6) asked about differences between Cypress and Scout.
SOC
استنادا إلى تصنيف SOC المهني
name evals-write-spec disable-model-invocation true description Write LLM evaluation spec files with datasets, tasks, and evaluators using the @kbn/evals Playwright fixture. Use when authoring new eval specs, adding datasets or evaluators, or debugging evaluation test failures.
Write Eval Specs
Spec File Anatomy
Eval specs use the evaluate Playwright fixture (not test). A spec file follows this structure:
import { evaluate, tags, selectEvaluators, type Example , type TaskOutput } from '@kbn/evals' ;
evaluate.describe ('Suite name' , { tag : tags.serverless .observability .complete }, () => {
evaluate.beforeAll (async ({ fetch, log }) => {
});
evaluate.afterAll (async ({ fetch, log }) => {
});
evaluate
'test name'
async
await
runExperiment
datasets
When a suite has a custom src/evaluate.ts, import from there instead of @kbn/evals:
import { evaluate } from '../src/evaluate' ;
Tags Every evaluate.describe must have a tag. Common choices:
Tag When to use tags.serverless.observability.completeObservability domain evals tags.serverless.security.completeSecurity domain evals tags.serverless.searchSearch domain evals tags.stateful.classicStateful-only evals
Import tags from @kbn/scout or @kbn/evals (re-exported).
Datasets A dataset is an array of examples with typed input, output (expected), and optional metadata:
type MyExample = Example <
{ question : string },
{ expectedAnswer : string },
{ tags ?: string [] }
>;
const dataset = {
name : 'my-dataset' ,
description : 'What this dataset tests' ,
examples : [
{
input : { question : 'What is 2+2?' },
output : { expectedAnswer : '4' },
metadata : { tags : ['math' ] },
},
],
};
Keep datasets focused. For local iteration, use --grep to run a subset:
node scripts/evals start --grep "my test name"
Tasks The task function receives an example and returns the output to evaluate:
task : async ({ input }) => {
const result = await someKibanaApi (input.question );
return { answer : result.content };
}
Tasks can use any fixture available in the evaluate callback: fetch, inferenceClient, connector, esClient, kbnClient, or custom fixtures like chatClient.
Evaluators There are two ways to provide evaluators to runExperiment:
Inline array -- pass evaluator objects directly (simple suites)selectEvaluatorsExample/TaskOutput generics
CODE Evaluators Deterministic, no LLM call. Use for binary checks:
{
name : 'NonEmpty' ,
kind : 'CODE' ,
evaluate : async ({ output }) => ({
score : output?.documents ?.length > 0 ? 1 : 0 ,
}),
}
LLM-as-Judge Criteria Use evaluators.criteria(criteriaArray) for subjective quality checks. The judge LLM scores each criterion:
evaluators.criteria ([
'The response correctly identifies the top users.' ,
'The response includes risk scores.' ,
]).evaluate ({ input, output, expected, metadata })
Correctness Analysis Compares output against expected answer:
evaluators.correctnessAnalysis ().evaluate ({ input, output, expected, metadata })
Groundedness Analysis Checks if output is grounded in provided context:
evaluators.groundednessAnalysis ().evaluate ({ input, output, expected, metadata })
Trace-Based Evaluators Available from evaluators.traceBasedEvaluators:
inputTokens, outputTokens, cachedTokens -- token usagetoolCalls -- number of tool callslatency -- span latency in seconds These read from the tracing ES cluster and require EDOT to be running.
RAG Evaluators For retrieval-augmented generation with ground truth:
import { createPrecisionAtKEvaluator, createRecallAtKEvaluator, createF1AtKEvaluator } from '@kbn/evals' ;
Available Fixtures Fixture Scope Description executorClientworker Runs experiments, exports scores to ES inferenceClientworker Inference REST client bound to connector connectorworker The model connector being evaluated evaluationConnectorworker The judge connector evaluatorsworker DefaultEvaluators (criteria, correctness, groundedness, trace-based)fetchworker HttpHandler for Kibana API callsesClientworker Elasticsearch client (Scout cluster) kbnClientworker Kibana client with retries traceEsClientworker ES client for trace queries evaluationsEsClientworker ES client for evaluation score storage logworker ToolingLog for structured loggingrepetitionsworker Number of experiment repetitions configworker Scout server config (hosts, auth)
The evaluateDataset Pattern For suites with many specs that share the same task + evaluator wiring, extract a reusable helper:
import type { DefaultEvaluators , EvalsExecutorClient } from '@kbn/evals' ;
import type { MyChatClient } from './chat_client' ;
export type EvaluateDataset = (opts : {
dataset: { name: string ; description: string ; examples: MyExample[] };
} Promise <void >;
export function createEvaluateDataset ({
chatClient, evaluators, executorClient,
}: {
chatClient: MyChatClient;
evaluators: DefaultEvaluators;
executorClient: EvalsExecutorClient;
} ): EvaluateDataset {
return async ({ dataset }) => {
await executorClient.runExperiment (
{
datasets : [dataset],
task : async ({ input }) => {
const response = await chatClient.converse ({ messages : [{ message : input.question }] });
return { messages : response.messages , steps : response.steps };
},
},
[myCriteriaEvaluator, myToolCallsEvaluator]
);
};
}
import { evaluate as base } from '../src/evaluate' ;
import type { EvaluateDataset } from '../src/evaluate_dataset' ;
import { createEvaluateDataset } from '../src/evaluate_dataset' ;
const evaluate = base.extend <{ evaluateDataset : EvaluateDataset }, {}>({
evaluateDataset : [
({ chatClient, evaluators, executorClient }, use ) => {
use (createEvaluateDataset ({ chatClient, evaluators, executorClient }));
},
{ scope : 'test' },
],
});
evaluate.describe ('My suite' , { tag : tags.serverless .search }, () => {
evaluate ('my test' , async ({ evaluateDataset }) => {
await evaluateDataset ({ dataset : { name : '...' , description : '...' , examples : [...] } });
});
});
Setup and Teardown Use evaluate.beforeAll / evaluate.afterAll for expensive one-time operations:
Install product docs : POST to /internal/product_doc_base/installCreate agents/rules : Use fetch or kbnClientLoad ES archives : Use esArchiver.load(archivePath) (requires custom fixture) Always clean up in afterAll -- delete agents, uninstall docs, unload archives.
Running Locally
node scripts/evals start
node scripts/evals start --model <connector-id> --judge <connector-id>
node scripts/evals start --grep "my test name"
node scripts/evals run --model <connector-id> --judge <connector-id>
Common Mistakes
Forgetting the tag on evaluate.describe -- Scout validates tags at runtime.
Missing afterAll cleanup -- leftover agents/docs pollute subsequent runs.
Overly large datasets for local iteration -- use --grep to target a single evaluate() block.
Importing evaluate from @kbn/evals when the suite has a custom src/evaluate.ts -- you'll miss custom fixtures.
Using test instead of evaluate -- the evaluate fixture provides all the evals-specific wiring.
References 