lai-gen-evals

name	lai-gen-evals
description	Expand and refine the evaluation suite for a Langium DSL project. Generates comprehensive eval files that cover syntactic correctness, semantic validity, user intent matching, edge cases, and language understanding.
user-invocable	true

Generate and Expand Evaluations

This skill guides the creation and expansion of a comprehensive evaluation suite for a Langium DSL project. A good eval suite measures not just whether an LLM can produce syntactically valid code, but whether it generates semantically correct programs that match user intent across a wide range of prompts.

You may also use the lai and langium skills for deeper understanding of the CLI workflow and Langium project structure.

When to Use

• Starter eval exists but is minimal — lai init generated a basic.eval.ts with one or two placeholder cases and you need real coverage
• No eval suite yet — the project has not been initialized; run lai init first to set up the evals directory and template files
• Eval pass rates are high but coverage is shallow — everything passes because you're only testing trivial cases
• Adding new language features — grammar or validator changes need matching eval cases
• LLM produces valid but wrong code — syntactically correct output that doesn't match what the user asked for
• Preparing for model comparison — building a thorough eval matrix to compare providers or models

Prerequisites

1. lai init completed — a lai.config.jsonc exists and an evals/ directory has been created. If not, run lai init first.
2. evals/utils.ts configured — the generateResponse() function must be wired to an LLM provider (OpenAI, Anthropic, Ollama, etc.). Check that the placeholder throw has been replaced.
3. A language descriptor and system prompt — generate these first with the lai-gen-descriptor and lai-gen-sysprompt skills if they don't exist.
4. A working Langium grammar with generated TypeScript artifacts and a create<Name>Services function.

Evaluation Architecture

File Organization

Evaluation files live in the evals/ directory (configured in lai.config.jsonc) and must have the .eval.ts extension. Organize them by evaluation category:

evals/
  utils.ts                    # shared LLM call helper (already generated by lai init)
  syntax.eval.ts              # syntactic correctness tests
  semantics.eval.ts           # semantic validity tests
  intent.eval.ts              # user intent matching tests
  edge-cases.eval.ts          # boundary conditions and tricky inputs
  understanding.eval.ts       # language comprehension tests
  advanced.eval.ts            # complex multi-feature programs

Each file can contain multiple describe() suites, each with multiple evaluation() cases. Keep files focused on a single evaluation category so results are easy to analyze.

The Evaluation API

All eval files use the langium-ai-tools/evals API:

import { describe, evaluation, beforeEach } from 'langium-ai-tools/evals';
import { LangiumEvaluator } from 'langium-ai-tools/evaluator';
import type { EvalContext } from 'langium-ai-tools/evals';
import { generateResponse, extractCodeBlock, calculateSimilarity } from './utils';
import { EmptyFileSystem } from 'langium';

Key concepts:

• ctx.systemPrompt — the generated system prompt, automatically injected into every eval
• LangiumEvaluator — validates generated code against the real Langium parser and validator
• score — a number between 0 and 1 returned from every evaluation (1 = perfect, 0 = complete failure)
• evaluation.each([...]) — parametrize a single eval across many inputs

Scoring Strategy

Every evaluation must return { score: number, ...extraData }. Design scores to be informative:

• Binary (0 or 1) — for hard pass/fail checks like "does it parse?"
• Continuous (0-1) — for graded checks like similarity, partial correctness
• Weighted composite — combine multiple signals: syntaxScore * 0.4 + semanticScore * 0.3 + intentScore * 0.3

Include extra data fields for debugging: diagnostics, similarity, matchedKeywords, responseLength, etc.

Step 1: Analyze the Language

Before writing evals, understand what needs to be tested. Read:

1. The grammar file — identify entry rules, key constructs, alternatives, cross-references, and optional elements
2. The validator — extract every validation check and its error message; each check should have at least one eval case that triggers it and one that avoids it
3. The scope provider — understand name resolution rules; these drive cross-reference evals
4. Existing examples — these are your ground truth for "correct" programs
5. Existing tests — the project's unit tests reveal expected behaviors and edge cases

Group what you find into evaluation categories (see Step 2).

Step 2: Design Evaluation Categories

A comprehensive eval suite should cover these categories. Not all apply to every language — include only what's relevant.

Category 1: Syntactic Correctness

Tests whether the LLM produces code that parses without errors. This is the baseline — if syntax fails, nothing else matters.

describe('Syntactic Correctness', () => {

    beforeEach(async () => {
        await services.shared.workspace.WorkspaceManager.initializeWorkspace([]);
    });

    // minimal valid program
    evaluation('generates a minimal valid program', async (ctx: EvalContext) => {
        const response = await generateResponse(
            'Generate the smallest possible valid program in this language.',
            { systemPrompt: ctx.systemPrompt }
        );
        const code = extractCodeBlock(response) || response;
        const result = await evaluator.evaluate(code);
        const score = (!result.data.failures && !result.data.errors) ? 1 : 0;
        return { score, ...result.data };
    });

    // parametrized across key constructs
    evaluation.each([
        { construct: 'a variable declaration', hint: 'Declare a single variable with a type' },
        { construct: 'a function definition', hint: 'Define a function that takes parameters and returns a value' },
        { construct: 'an import statement', hint: 'Import a module or type from another file' },
        // add one entry per major grammar construct in your language
    ])('generates valid syntax for $construct', (data) => async (ctx: EvalContext) => {
        const response = await generateResponse(
            `Generate a valid program that contains ${data.construct}. ${data.hint}`,
            { systemPrompt: ctx.systemPrompt }
        );
        const code = extractCodeBlock(response) || response;
        const result = await evaluator.evaluate(code);
        const score = (!result.data.failures && !result.data.errors) ? 1 : 0;
        return { score, construct: data.construct, ...result.data };
    });
});

What to parametrize: one entry for each major grammar construct — entry rules, declarations, expressions, statements, type annotations, modifiers, etc. Derive these directly from the grammar's parser rules.

Category 2: Semantic Validity

Tests whether generated code passes the validator — no errors, no unresolved references, correct types. Syntax can be valid while semantics are broken.

describe('Semantic Validity', () => {

    beforeEach(async () => {
        await services.shared.workspace.WorkspaceManager.initializeWorkspace([]);
    });

    // cross-references resolve correctly
    evaluation('generates valid cross-references', async (ctx: EvalContext) => {
        const response = await generateResponse(
            'Generate a program where one element references another by name. ' +
            'Ensure the referenced element is declared before it is used.',
            { systemPrompt: ctx.systemPrompt }
        );
        const code = extractCodeBlock(response) || response;
        const result = await evaluator.evaluate(code);
        // check specifically for linking errors (unresolved references)
        const linkErrors = result.data.diagnostics.filter(
            d => d.message.includes('Could not resolve reference')
        );
        const score = linkErrors.length === 0 && !result.data.failures ? 1 : 0;
        return { score, linkErrors: linkErrors.length, ...result.data };
    });

    // validator-specific constraints
    evaluation.each([
        {
            rule: 'unique names',
            prompt: 'Generate a program with multiple declarations, each having a unique name.',
        },
        {
            rule: 'required fields',
            prompt: 'Generate a complete entity with all required fields filled in.',
        },
        // add one entry per key validation rule
    ])('respects validation rule: $rule', (data) => async (ctx: EvalContext) => {
        const response = await generateResponse(data.prompt, {
            systemPrompt: ctx.systemPrompt
        });
        const code = extractCodeBlock(response) || response;
        const result = await evaluator.evaluate(code);
        const errors = result.data.diagnostics.filter(d => d.severity === 1);
        const score = errors.length === 0 && !result.data.failures ? 1 : 0;
        return { score, rule: data.rule, errorCount: errors.length, ...result.data };
    });
});

What to parametrize: one entry for each validation check in the validator source. Read the validator's check* methods and translate each into a prompt that should produce code satisfying that constraint.

Category 3: User Intent Matching

Tests whether the LLM generates code that does what the user asked — not just valid code, but the right code. This is where eval quality separates good suites from trivial ones.

describe('User Intent Matching', () => {

    beforeEach(async () => {
        await services.shared.workspace.WorkspaceManager.initializeWorkspace([]);
    });

    // exact expected output
    evaluation.each([
        {
            prompt: 'Create a Person entity with name (string) and age (number) fields',
            expected: 'entity Person {\n  name: string\n  age: number\n}',
            keywords: ['Person', 'name', 'string', 'age', 'number'],
        },
        {
            prompt: 'Create an empty Order entity',
            expected: 'entity Order {\n}',
            keywords: ['Order'],
        },
        // add cases derived from your language's examples and common use patterns
    ])('matches intent: $prompt', (data) => async (ctx: EvalContext) => {
        const response = await generateResponse(data.prompt, {
            systemPrompt: ctx.systemPrompt
        });
        const code = extractCodeBlock(response) || response;
        const result = await evaluator.evaluate(code);

        // syntax + semantics must pass
        const isValid = !result.data.failures && !result.data.errors
            && result.data.diagnostics.length === 0;

        // check structural intent via keywords
        let keywordHits = 0;
        const lowerCode = code.toLowerCase();
        for (const kw of data.keywords) {
            if (lowerCode.includes(kw.toLowerCase())) {
                keywordHits++;
            }
        }
        const keywordScore = data.keywords.length > 0
            ? keywordHits / data.keywords.length
            : 1;

        // optional: text similarity to expected output
        const similarity = calculateSimilarity(code, data.expected);

        // composite score: validity is a gate, then blend keyword + similarity
        const score = isValid
            ? keywordScore * 0.5 + similarity * 0.5
            : 0;

        return {
            score,
            isValid,
            keywordScore,
            similarity,
            ...result.data,
        };
    });
});

What to parametrize: build cases from:

• Every example file in the descriptor — reverse-engineer a natural language prompt from the code, then use the code as the expected output
• Common user tasks in the language's domain
• Prompts of varying specificity (vague vs. precise) to test robustness

Category 4: Edge Cases and Error Boundaries

Tests tricky inputs, boundary conditions, and prompts that might trip up the LLM. These cases probe for subtle failures.

describe('Edge Cases', () => {

    beforeEach(async () => {
        await services.shared.workspace.WorkspaceManager.initializeWorkspace([]);
    });

    // empty or near-empty programs
    evaluation('handles request for empty program', async (ctx: EvalContext) => {
        const response = await generateResponse(
            'Generate a valid but completely empty program (if the language allows it), ' +
            'or the smallest possible program.',
            { systemPrompt: ctx.systemPrompt }
        );
        const code = extractCodeBlock(response) || response;
        const result = await evaluator.evaluate(code);
        const score = !result.data.failures && !result.data.errors ? 1 : 0;
        return { score, ...result.data };
    });

    // special characters and unicode
    evaluation('handles special characters in identifiers', async (ctx: EvalContext) => {
        const response = await generateResponse(
            'Generate a program using identifiers that contain underscores or numbers ' +
            '(e.g., my_var_2, item_01). Ensure the program is syntactically valid.',
            { systemPrompt: ctx.systemPrompt }
        );
        const code = extractCodeBlock(response) || response;
        const result = await evaluator.evaluate(code);
        const score = !result.data.failures && !result.data.errors ? 1 : 0;
        return { score, ...result.data };
    });

    // deeply nested structures (if the language supports nesting)
    evaluation('handles deeply nested structures', async (ctx: EvalContext) => {
        const response = await generateResponse(
            'Generate a program with 3 or more levels of nesting ' +
            '(e.g., nested blocks, nested types, or nested expressions).',
            { systemPrompt: ctx.systemPrompt }
        );
        const code = extractCodeBlock(response) || response;
        const result = await evaluator.evaluate(code);
        const score = !result.data.failures && !result.data.errors ? 1 : 0;
        return { score, ...result.data };
    });

    // large programs
    evaluation('generates a large valid program', async (ctx: EvalContext) => {
        const response = await generateResponse(
            'Generate a large, realistic program with at least 5 different declarations ' +
            'and multiple cross-references between them.',
            { systemPrompt: ctx.systemPrompt }
        );
        const code = extractCodeBlock(response) || response;
        const result = await evaluator.evaluate(code);
        const score = !result.data.failures && !result.data.errors ? 1 : 0;
        return { score, ...result.data };
    });

    // ambiguous prompts
    evaluation('handles ambiguous prompts gracefully', async (ctx: EvalContext) => {
        const response = await generateResponse(
            'Generate some code.',
            { systemPrompt: ctx.systemPrompt }
        );
        const code = extractCodeBlock(response) || response;
        const result = await evaluator.evaluate(code);
        // just needs to produce something parseable
        const score = !result.data.failures && !result.data.errors ? 1 : 0;
        return { score, ...result.data };
    });
});

What to include: derive edge cases from the grammar's optional rules, recursive rules, and the validator's negative test cases. Also consider:

• Keywords used as identifiers (if the language allows it)
• Maximum/minimum cardinality of lists
• Optional clauses present vs. absent
• Combinations of features that rarely appear together

Category 5: Language Understanding

Tests whether the LLM can explain and reason about DSL code, not just generate it. This validates comprehension beyond syntax production.

describe('Language Understanding', () => {

    // explanation quality
    evaluation.each([
        {
            name: 'basic program',
            code: '/* paste a simple example program here */',
            expectedConcepts: ['entity', 'property', 'type'],
        },
        {
            name: 'complex program',
            code: '/* paste a complex example program here */',
            expectedConcepts: ['inheritance', 'reference', 'validation'],
        },
    ])('explains $name correctly', (data) => async (ctx: EvalContext) => {
        const response = await generateResponse(
            `Explain what this code does and identify the key language concepts used:\n\`\`\`\n${data.code}\n\`\`\``,
            { systemPrompt: ctx.systemPrompt }
        );
        const lowerResponse = response.toLowerCase();
        let hits = 0;
        for (const concept of data.expectedConcepts) {
            if (lowerResponse.includes(concept.toLowerCase())) {
                hits++;
            }
        }
        const score = data.expectedConcepts.length > 0
            ? hits / data.expectedConcepts.length
            : 1;
        return { score, matchedConcepts: hits, totalConcepts: data.expectedConcepts.length };
    });

    // error diagnosis
    evaluation('diagnoses a syntax error correctly', async (ctx: EvalContext) => {
        // TODO: paste a program with a deliberate syntax error
        const brokenCode = '/* intentionally broken code here */';
        const response = await generateResponse(
            `This code has an error. Identify the problem and explain how to fix it:\n\`\`\`\n${brokenCode}\n\`\`\``,
            { systemPrompt: ctx.systemPrompt }
        );
        // check if the response identifies the error area
        const mentionsError = response.toLowerCase().includes('error')
            || response.toLowerCase().includes('missing')
            || response.toLowerCase().includes('invalid')
            || response.toLowerCase().includes('unexpected');
        const score = mentionsError ? 1 : 0;
        return { score };
    });

    // modification tasks
    evaluation('modifies existing code correctly', async (ctx: EvalContext) => {
        // TODO: paste a valid program and ask for a specific modification
        const originalCode = '/* valid program here */';
        const response = await generateResponse(
            `Given this existing code:\n\`\`\`\n${originalCode}\n\`\`\`\n` +
            'Add a new field called "email" of type "string" to the Person entity.',
            { systemPrompt: ctx.systemPrompt }
        );
        const code = extractCodeBlock(response) || response;
        const result = await evaluator.evaluate(code);
        const isValid = !result.data.failures && !result.data.errors
            && result.data.diagnostics.length === 0;
        const hasEmail = code.toLowerCase().includes('email');
        const score = isValid && hasEmail ? 1 : (isValid ? 0.5 : 0);
        return { score, isValid, hasEmail, ...result.data };
    });
});

Category 6: Advanced / Multi-Feature Programs

Tests the LLM's ability to combine multiple language features in a single program. These are harder prompts that reflect realistic usage.

describe('Advanced Programs', () => {

    beforeEach(async () => {
        await services.shared.workspace.WorkspaceManager.initializeWorkspace([]);
    });

    evaluation.each([
        {
            scenario: 'full domain model',
            prompt: 'Generate a complete domain model for an online bookstore with at least 3 entities ' +
                    '(Book, Author, Order) that reference each other.',
            requiredElements: ['Book', 'Author', 'Order'],
        },
        {
            scenario: 'inheritance hierarchy',
            prompt: 'Generate a type hierarchy where a base type is extended by at least 2 subtypes, ' +
                    'each adding their own properties.',
            requiredElements: [],
        },
        // add scenarios that combine features specific to your language
    ])('generates $scenario', (data) => async (ctx: EvalContext) => {
        const response = await generateResponse(data.prompt, {
            systemPrompt: ctx.systemPrompt,
            maxTokens: 4096,
        });
        const code = extractCodeBlock(response) || response;
        const result = await evaluator.evaluate(code);

        const isValid = !result.data.failures && !result.data.errors
            && result.data.diagnostics.length === 0;

        let elementHits = 0;
        const lowerCode = code.toLowerCase();
        for (const el of data.requiredElements) {
            if (lowerCode.includes(el.toLowerCase())) {
                elementHits++;
            }
        }
        const elementScore = data.requiredElements.length > 0
            ? elementHits / data.requiredElements.length
            : 1;

        const score = isValid ? elementScore : 0;
        return { score, isValid, elementScore, ...result.data };
    });
});

Step 3: Generate the Eval Files

For each category from Step 2 that applies to your language, create a .eval.ts file in the evals/ directory. Follow this process:

1. Read the grammar to identify all major constructs — each becomes a parametrized test case in the Syntactic Correctness suite
2. Read the validator to extract every check — each becomes a case in the Semantic Validity suite
3. Read the examples from the descriptor — reverse-engineer prompts from each example and use the example code as expected output for the Intent Matching suite
4. Read the tests for edge cases — negative tests and boundary tests become the Edge Cases suite
5. Derive understanding tests from the examples — use real code snippets for explanation and modification tasks

Common Setup Pattern

Every eval file starts with the same boilerplate:

import { describe, evaluation, beforeEach } from 'langium-ai-tools/evals';
import { LangiumEvaluator } from 'langium-ai-tools/evaluator';
import type { EvalContext } from 'langium-ai-tools/evals';
import { generateResponse, extractCodeBlock, calculateSimilarity } from './utils';
import { EmptyFileSystem } from 'langium';
// REPLACE: import your DSL's service creation function
import { create__DSL_NAME__Services } from '../src/__module_path__';

const services = create__DSL_NAME__Services(EmptyFileSystem).__ServiceKey__;
const evaluator = new LangiumEvaluator(services);

Replace the placeholders with your project's actual service creation function, module path, and service key (same values used in the MCP server template).

Helper: Validity Check

To avoid repeating the same validity check in every eval, define a helper in your eval file or in utils.ts:

function isClean(result: { data: { failures: number; errors: number; diagnostics: { severity?: number }[] } }): boolean {
    return !result.data.failures && !result.data.errors && result.data.diagnostics.length === 0;
}

Step 4: Run and Iterate

After generating eval files:

# run all evaluations
lai evaluate

# verbose output to see scores and diagnostics per case
lai evaluate --verbose

# view results
lai show latest --verbose

# tag this as your baseline
lai tag latest baseline-coverage

Interpreting Results

• High syntax scores, low intent scores — the system prompt teaches valid syntax but not enough domain context. Refine the descriptor with more examples and documentation.
• Low syntax scores across the board — the system prompt is missing grammar details. Regenerate with lai gen sysprompt --fresh after enriching the descriptor.
• Some constructs always fail — the system prompt doesn't cover that grammar construct. Add an example that uses it.
• Edge cases all fail — expected for initial runs. Use failures to refine the system prompt's "Common Mistakes" section.

Expanding Over Time

As the language evolves:

1. Add eval cases for new grammar rules and validation checks
2. Convert bug reports into regression eval cases
3. Add cases for prompts that real users submit
4. Periodically review low-scoring cases — if a case consistently scores < 0.3 despite prompt refinements, consider whether the prompt is unreasonable or whether the system prompt needs a specific addition

Coverage Checklist

Use this checklist to assess whether your eval suite is comprehensive:

• Every major grammar construct has at least one syntax eval
• Every validation rule has a positive eval (code that passes) and ideally a negative eval (prompt that should avoid triggering the error)
• Every example in the descriptor has a corresponding intent-matching eval
• Cross-references are tested with at least one valid and one multi-reference case
• Edge cases cover: empty/minimal programs, large programs, deep nesting, special characters, ambiguous prompts
• Understanding evals cover: explanation, error diagnosis, code modification
• Advanced evals combine at least 2-3 language features per case
• Parametrized tests are used wherever multiple inputs share the same evaluation logic