Skip to main content
evals-write-spec Write LLM evaluation spec files with datasets, tasks, and evaluators using the @kbn/evals Playwright fixture. Use when authoring new eval specs, adding datasets or evaluators, or debugging evaluation test failures.
Aller à l'installation Skills Marketplace Découvrez et explorez les compétences IA créées par la communauté.
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Afficher les détails du prompt Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
npx skills add https://github.com/elastic/kibana --skill evals-write-specLa commande reste sur une seule ligne. Faites défiler horizontalement pour la vérifier avant de la copier.
Vous préférez une copie locale ? Téléchargez les fichiers actuellement disponibles dans SkillsMP.
Télécharger Zip Téléchargement... SOC
Basé sur la classification professionnelle SOC
cypress-to-scout-migration Migrate Kibana Cypress E2E tests (.cy.ts) to Scout (Playwright). Applies to any Kibana plugin or solution. Includes triage gates (duplicate detection, layer analysis, value assessment), Cypress-to-Scout pattern mapping, data cleanup audit, and PR workflow. Use when: (1) migrating a Cypress test to Scout, (2) converting .cy.ts to .spec.ts, (3) planning a Cypress-to-Scout migration batch, (4) rewriting Cypress screens/tasks as Scout page objects, (5) asked "how do I move this Cypress test to Scout/Playwright", (6) asked about differences between Cypress and Scout.
Explorateur de fichiers
3 fichiers name evals-write-spec disable-model-invocation true description Write LLM evaluation spec files with datasets, tasks, and evaluators using the @kbn/evals Playwright fixture. Use when authoring new eval specs, adding datasets or evaluators, or debugging evaluation test failures.
Write Eval Specs
Spec File Anatomy
Eval specs use the evaluate Playwright fixture (not test). A spec file follows this structure:
import { evaluate, tags, selectEvaluators, type Example , type TaskOutput } from '@kbn/evals' ;
evaluate.describe ('Suite name' , { tag : tags.serverless .observability .complete }, () => {
evaluate.beforeAll (async ({ fetch, log }) => {
});
evaluate.afterAll (async ({ fetch, log }) => {
});
evaluate
'test name'
async
await
runExperiment
datasets
When a suite has a custom src/evaluate.ts, import from there instead of @kbn/evals:
import { evaluate } from '../src/evaluate' ;
Tags Every evaluate.describe must have a tag. Common choices:
Tag When to use tags.serverless.observability.completeObservability domain evals tags.serverless.security.completeSecurity domain evals tags.serverless.searchSearch domain evals tags.stateful.classicStateful-only evals
Import tags from @kbn/scout or @kbn/evals (re-exported).
Datasets A dataset is an array of examples with typed input, output (expected), and optional metadata:
type MyExample = Example <
{ question : string },
{ expectedAnswer : string },
{ tags ?: string [] }
>;
const dataset = {
name : 'my-dataset' ,
description : 'What this dataset tests' ,
examples : [
{
input : { question : 'What is 2+2?' },
output : { expectedAnswer : '4' },
metadata : { tags : ['math' ] },
},
],
};
Keep datasets focused. For local iteration, use --grep to run a subset:
node scripts/evals start --grep "my test name"
Tasks The task function receives an example and returns the output to evaluate:
task : async ({ input }) => {
const result = await someKibanaApi (input.question );
return { answer : result.content };
}
Tasks can use any fixture available in the evaluate callback: fetch, inferenceClient, connector, esClient, kbnClient, or custom fixtures like chatClient.
Evaluators There are two ways to provide evaluators to runExperiment:
Inline array -- pass evaluator objects directly (simple suites)selectEvaluatorsExample/TaskOutput generics
CODE Evaluators Deterministic, no LLM call. Use for binary checks:
{
name : 'NonEmpty' ,
kind : 'CODE' ,
evaluate : async ({ output }) => ({
score : output?.documents ?.length > 0 ? 1 : 0 ,
}),
}
LLM-as-Judge Criteria Use evaluators.criteria(criteriaArray) for subjective quality checks. The judge LLM scores each criterion:
evaluators.criteria ([
'The response correctly identifies the top users.' ,
'The response includes risk scores.' ,
]).evaluate ({ input, output, expected, metadata })
Correctness Analysis Compares output against expected answer:
evaluators.correctnessAnalysis ().evaluate ({ input, output, expected, metadata })
Groundedness Analysis Checks if output is grounded in provided context:
evaluators.groundednessAnalysis ().evaluate ({ input, output, expected, metadata })
Trace-Based Evaluators Available from evaluators.traceBasedEvaluators:
inputTokens, outputTokens, cachedTokens -- token usagetoolCalls -- number of tool callslatency -- span latency in seconds These read from the tracing ES cluster and require EDOT to be running.
RAG Evaluators For retrieval-augmented generation with ground truth:
import { createPrecisionAtKEvaluator, createRecallAtKEvaluator, createF1AtKEvaluator } from '@kbn/evals' ;
Available Fixtures Fixture Scope Description executorClientworker Runs experiments, exports scores to ES inferenceClientworker Inference REST client bound to connector connectorworker The model connector being evaluated evaluationConnectorworker The judge connector evaluatorsworker DefaultEvaluators (criteria, correctness, groundedness, trace-based)fetchworker HttpHandler for Kibana API callsesClientworker Elasticsearch client (Scout cluster) kbnClientworker Kibana client with retries traceEsClientworker ES client for trace queries evaluationsEsClientworker ES client for evaluation score storage logworker ToolingLog for structured loggingrepetitionsworker Number of experiment repetitions configworker Scout server config (hosts, auth)
The evaluateDataset Pattern For suites with many specs that share the same task + evaluator wiring, extract a reusable helper:
import type { DefaultEvaluators , EvalsExecutorClient } from '@kbn/evals' ;
import type { MyChatClient } from './chat_client' ;
export type EvaluateDataset = (opts : {
dataset: { name: string ; description: string ; examples: MyExample[] };
} Promise <void >;
export function createEvaluateDataset ({
chatClient, evaluators, executorClient,
}: {
chatClient: MyChatClient;
evaluators: DefaultEvaluators;
executorClient: EvalsExecutorClient;
} ): EvaluateDataset {
return async ({ dataset }) => {
await executorClient.runExperiment (
{
datasets : [dataset],
task : async ({ input }) => {
const response = await chatClient.converse ({ messages : [{ message : input.question }] });
return { messages : response.messages , steps : response.steps };
},
},
[myCriteriaEvaluator, myToolCallsEvaluator]
);
};
}
import { evaluate as base } from '../src/evaluate' ;
import type { EvaluateDataset } from '../src/evaluate_dataset' ;
import { createEvaluateDataset } from '../src/evaluate_dataset' ;
const evaluate = base.extend <{ evaluateDataset : EvaluateDataset }, {}>({
evaluateDataset : [
({ chatClient, evaluators, executorClient }, use ) => {
use (createEvaluateDataset ({ chatClient, evaluators, executorClient }));
},
{ scope : 'test' },
],
});
evaluate.describe ('My suite' , { tag : tags.serverless .search }, () => {
evaluate ('my test' , async ({ evaluateDataset }) => {
await evaluateDataset ({ dataset : { name : '...' , description : '...' , examples : [...] } });
});
});
Setup and Teardown Use evaluate.beforeAll / evaluate.afterAll for expensive one-time operations:
Install product docs : POST to /internal/product_doc_base/installCreate agents/rules : Use fetch or kbnClientLoad ES archives : Use esArchiver.load(archivePath) (requires custom fixture) Always clean up in afterAll -- delete agents, uninstall docs, unload archives.
Running Locally
node scripts/evals start
node scripts/evals start --model <connector-id> --judge <connector-id>
node scripts/evals start --grep "my test name"
node scripts/evals run --model <connector-id> --judge <connector-id>
Common Mistakes
Forgetting the tag on evaluate.describe -- Scout validates tags at runtime.
Missing afterAll cleanup -- leftover agents/docs pollute subsequent runs.
Overly large datasets for local iteration -- use --grep to target a single evaluate() block.
Importing evaluate from @kbn/evals when the suite has a custom src/evaluate.ts -- you'll miss custom fixtures.
Using test instead of evaluate -- the evaluate fixture provides all the evals-specific wiring.
References 