eval-writer

# Eval Writer Create new eval suites for the `deepagentsjs` monorepo. Each eval is an independent workspace package under `evals/` that uses the `@deepagents/evals` harness, runs via vitest, and reports results to LangSmith. ## Before you start Read the existing eval infrastructure to understand current patterns: ``` internal/eval-harness/src/index.ts # EvalRunner, RunAgentParams, matchers internal/eval-harness/src/deepagent.ts # DeepAgentEvalRunner, extend() internal/eval-harness/src/setup.ts # Registered runners evals/README.md # User-facing docs internal/eval-harness/README.md # Harness internals ``` Scan existing evals for conventions: ``` evals/basic/index.test.ts # Simple: system prompt, reasoning evals/files/index.test.ts # File ops: read, write, edit, glob, grep evals/subagents/index.test.ts # Delegation: task tool, named subagents ``` ## Workflow ### 1. Understand the eval requirements Clarify with the user: - **What capability** is being evaluated? (file ops, tool use, multi-turn reasoning, memory, code generation, etc.) - **Where do test cases come from?** Options: - **Inline** — hardcoded in the test file (simple, good for <20 cases) - **JSON/JSONL fixture** — checked into the eval package (good for 20-200 cases) - **External dataset** — downloaded at setup time (good for published benchmarks) - **LangSmith dataset** — pulled from LangSmith API (good for collaborative curation) - **How should results be scored?** Options: - **Trajectory matchers** — step count, tool calls, final text (built-in) - **Exact/fuzzy match** — compare output to reference (simple) - **LLM-as-judge** — use a model to grade the output (complex evals) - **Code execution** — run generated code and check results (SWE-bench style) - **Custom evaluator** — domain-specific scoring function - **Does the agent need special configuration?** (custom tools, subagents, system prompt, initial files) ### 2. Create the eval package Every eval is a workspace package under `evals/<name>/`. #### Directory structure ``` evals/<name>/ ├── package.json ├── vitest.config.ts ├── index.test.ts ├── README.md └── (optional) fixtures/ # JSON/JSONL test data └── (optional) vitest.setup.ts # Dataset loading, custom setup └── (optional) evaluators.ts # Custom scoring functions ``` #### package.json ```json { "name": "@deepagents/eval-<name>", "private": true, "type": "module", "scripts": { "test:eval": "vitest run" }, "dependencies": { "@deepagents/evals": "workspace:*", "deepagents": "workspace:*", "langsmith": "^0.5.4", "vitest": "^4.0.18" } } ``` Add extra dependencies as needed (e.g. `zod` for tool schemas, `langchain` for `tool()` helper, dataset-specific packages). #### vitest.config.ts ```ts import { defineConfig } from "vitest/config"; export default defineConfig({ test: { environment: "node", globals: false, testTimeout: 120_000, hookTimeout: 60_000, teardownTimeout: 60_000, include: ["**/*.test.ts"], setupFiles: ["@deepagents/evals/setup"], reporters: ["default", "langsmith/vitest/reporter"], }, }); ``` Adjust `testTimeout` for long-running evals (multi-turn, code execution). Add `"./vitest.setup.ts"` to `setupFiles` if the eval needs custom setup (dataset loading, etc.). #### README.md ```markdown # <name> <One-line description of what this eval tests.> ``` #### Verify workspace registration Check that `pnpm-workspace.yaml` includes `"evals/*"`. It should already be there — if not, add it. ### 3. Design test cases #### Pattern A: Inline test cases (simple evals) Best for small, hand-crafted test suites. Each test is an `ls.test()` call with `inputs` and optional `referenceOutputs`. ```ts ls.test( "descriptive test name", { inputs: { query: "What is 2+2?" }, referenceOutputs: { expectedAnswer: "4" }, }, async ({ inputs, referenceOutputs }) => { const result = await runner.run({ query: inputs.query }); // assertions... }, ); ``` #### Pattern B: Data-driven with ls.test.each (medium evals) Best for 10-200 cases from a fixture file. Load the data and iterate: ```ts import testCases from "./fixtures/cases.json"; // testCases = [{ inputs: { query: "..." }, referenceOutputs: { answer: "..." } }, ...] ls.test.each(testCases)( "case: ${inputs.query}", async ({ inputs, referenceOutputs }) => { const result = await runner.run({ query: inputs.query }); // assertions using referenceOutputs... }, ); ``` The fixture JSON must be an array of objects with at minimum `{ inputs: {...} }`. Optional fields: `referenceOutputs`, `id`, `metadata`, `split`. #### Pattern C: External dataset (published benchmarks) For published benchmarks (oolong, AgentBench, SWE-bench, etc.), download and cache the dataset in a setup file. Create `vitest.setup.ts`: ```ts import { existsSync, mkdirSync, writeFileSync, readFileSync } from "fs"; import { join } from "path"; const CACHE_DIR = join(import.meta.dirname, ".cache"); const DATA_PATH = join(CACHE_DIR, "dataset.json"); export async function loadDataset(): Promise<TestCase[]> { if (existsSync(DATA_PATH)) { return JSON.parse(readFileSync(DATA_PATH, "utf-8")); } mkdirSync(CACHE_DIR, { recursive: true }); // Download from source — adapt to the specific benchmark const response = await fetch("https://example.com/dataset.json"); const data = await response.json(); // Transform into eval format const cases = data.map((item: any) => ({ inputs: { query: item.question }, referenceOutputs: { answer: item.gold_answer }, metadata: { source: item.id, category: item.category }, })); writeFileSync(DATA_PATH, JSON.stringify(cases, null, 2)); return cases; } ``` Add `.cache/` to `.gitignore` in the eval package. Then register it as a vitest setup file in `vitest.config.ts`: ```ts setupFiles: ["@deepagents/evals/setup", "./vitest.setup.ts"], ``` And in the test file: ```ts import { loadDataset } from "./vitest.setup.js"; const dataset = await loadDataset(); ls.describe(runner.name, () => { ls.test.each(dataset)( "${metadata.source}: ${inputs.query}", async ({ inputs, referenceOutputs }) => { // ... }, ); }, { projectName: "deepagents-js-<name>", upsert: true }); ``` #### Pattern D: LangSmith dataset Pull test cases from a LangSmith dataset. Useful for collaborative curation where non-engineers add examples via the LangSmith UI. ```ts import { Client } from "langsmith"; const client = new Client(); export async function loadDataset(): Promise<TestCase[]> { const examples = []; for await (const example of client.listExamples({ datasetName: "my-dataset-name", })) { examples.push({ id: example.id, inputs: example.inputs, referenceOutputs: example.outputs ?? {}, }); } return examples; } ``` ### 4. Write scoring logic #### Built-in trajectory matchers The harness provides vitest matchers that also log LangSmith feedback scores. Use these as the primary building blocks: ```ts // Exact step count expect(result).toHaveAgentSteps(3); // Exact tool-call count across all steps expect(result).toHaveToolCallRequests(2); // Check a specific tool call in step N (1-indexed) expect(result).toHaveToolCallInStep(1, { name: "write_file", argsContains: { file_path: "/out.txt" }, // partial match argsEquals: { file_path: "/out.txt" }, // exact match }); // Final response text expect(result).toHaveFinalTextContaining("hello", true /* caseInsensitive */); // Extract final text for custom assertions import { getFinalText } from "@deepagents/evals"; const text = getFinalText(result); expect(text.trim()).toBe("4"); // File system assertions expect(result.files["/output.md"]).toContain("expected content"); expect(Object.keys(result.files)).toHaveLength(3); ``` #### Custom feedback logging Log additional LangSmith feedback scores beyond what matchers provide: ```ts import * as ls from "langsmith/vitest"; // Numeric score ls.logFeedback({ key: "accuracy", score: 0.95 }); // Boolean score ls.logFeedback({ key: "correct", score: 1 }); // With comment ls.logFeedback({ key: "quality", score: 0.8, comment: "Minor formatting issue" }); ``` #### LLM-as-judge evaluators For subjective quality, use `ls.wrapEvaluator()` to create a traced evaluator that logs feedback automatically: ```ts import * as ls from "langsmith/vitest"; import { ChatAnthropic } from "@langchain/anthropic"; const judge = new ChatAnthropic({ model: "claude-sonnet-4-5-20250929" }); const evaluateHelpfulness = ls.wrapEvaluator( async ({ inputs, outputs, referenceOutputs }) => { const response = await judge.invoke([ { role: "system", content: `Rate the helpfulness of the assistant's response on a scale of 0-1. Respond with JSON: { "score": <number>, "reasoning": "<explanation>" }`, }, { role: "user", content: `Question: ${inputs.query}\nExpected: ${referenceOutputs.answer}\nActual: ${outputs.response}`, }, ]); const parsed = JSON.parse(response.content as string); return { key: "helpfulness", score: parsed.score, comment: parsed.reasoning, }; }, ); // In a test: const result = await runner.run({ query: inputs.query }); const text = getFinalText(result); await evaluateHelpfulness({ inputs: { query: inputs.query }, outputs: { response: text }, referenceOutputs: referenceOutputs ?? {}, }); ``` ### 5. Wire up the test file #### Minimal template ```ts import * as ls from "langsmith/vitest"; import { expect } from "vitest"; import { getDefaultRunner, getFinalText } from "@deepagents/evals"; const runner = getDefaultRunner(); ls.describe( runner.name, () => { ls.test( "test name", { inputs: { query: "..." }, referenceOutputs: { answer: "..." }, }, async ({ inputs, referenceOutputs }) => { const result = await runner.run({ query: inputs.query }); expect(result).toHaveAgentSteps(1); expect(result).toHaveFinalTextContaining(referenceOutputs.answer); }, ); }, { projectName: "deepagents-js-<name>", upsert: true }, ); ``` #### Key conventions - **`getDefaultRunner()`** — reads `EVAL_RUNNER` env var. Throws if not set. - **`runner.name`** — used as `ls.describe` name → becomes the LangSmith dataset name. - **`runner.run({ query, initialFiles? })`** — pure invocation. Returns `AgentTrajectory`. - **`runner.extend({ systemPrompt?, tools?, subagents?, ... })`** — returns a new runner with agent config overrides. Use for tests that need custom agent setup.