Menu
Guides creation of synthetic eval cases that expand coverage without drifting away from real usage. Use when the current eval set is too small, too repetitive, or missing edge cases, and you need more diverse prompts, distractors, or structured scenarios.
Guides design of LLM-as-judge prompts for subjective evaluation criteria. Use when deterministic checks are insufficient and you need a judge prompt for quality dimensions like helpfulness, faithfulness, clarity, or tone.
Guides building or improving interfaces for human review of eval traces. Use when humans need to inspect failures, label outputs, compare model behavior, or audit evaluator decisions at scale.
Guides systematic analysis of eval failures using traces. Use when a suite is failing, model outputs seem inconsistent, evaluator behavior is suspect, or you need to classify failures before changing prompts, metrics, or datasets.
Guides evaluation of RAG pipeline retrieval and generation quality. Use when evaluating a retrieval-augmented generation system, measuring retrieval quality, assessing generation faithfulness or relevance, generating synthetic QA pairs for retrieval testing, or optimizing chunking strategies.
Drafts a reviewable judge-template patch from evaluator validation disagreements.
Guides validation of evaluators, especially LLM judges, against labeled examples. Use when evaluator quality is uncertain, judge scores seem inconsistent, or you need to check whether the evaluator is biased, noisy, or misaligned.
| name | generate-synthetic-data |
| description | Guides creation of synthetic eval cases that expand coverage without drifting away from real usage. Use when the current eval set is too small, too repetitive, or missing edge cases, and you need more diverse prompts, distractors, or structured scenarios. |
Inspect the current eval suite and recent traces first. Determine what dimensions are under-covered. Use evals/, traces/, and evaluators/index.mjs to understand what kinds of examples the repo can score well.
Synthetic data should feed the label workflow: generate inputs, run them through the actual system, review traces, store human labels, then promote durable examples into eval suites.
Define the dimensions that matter for the task, such as:
Generate examples by combining those dimensions intentionally.
Create combinations of task dimensions and ensure each combination is represented by at least one eval case. This prevents synthetic generation from overproducing the easiest examples.
A synthetic example is useful only if you can score it reliably. Prefer cases that support:
Do not generate cases whose “correct answer” is too vague to evaluate.
Reject examples that are:
Keep only synthetic data that expands meaningful coverage.
evals/dataset.mjsrun-eval.mjsevaluators/index.mjstraces/labels/docs/schemas/labels.md