# vitest-evals
Monorepo for the explicit-run `vitest-evals` shape:
- `packages/vitest-evals`: core suite API, judges, normalized harness/session
types, and reporter
- `packages/core`: shared primitives, JSON artifact schemas, and full-report
collection helpers
- `packages/report-ui`: local React SPA and CLI for inspecting Vitest JSON
report artifacts
- `packages/harness-ai-sdk`: `ai-sdk`-focused harness adapter
- `packages/harness-openai-agents`: `@openai/agents`-focused harness adapter
- `packages/harness-pi-ai`: `pi-ai`-focused harness adapter with tool replay
- `packages/github-reporter`: GitHub Actions summary, annotation, and optional
Check Run publishing from Vitest JSON output
- `apps/demo-pi`: end-to-end Pi Mono demo evals with an app-local refund agent
- `apps/demo-ai-sdk`: end-to-end AI SDK demo evals with app-local refund tools
- `apps/demo-openai-agents`: end-to-end OpenAI Agents demo evals with
app-local refund tools
## Reading Guide
- Start with the public docs site for the guided setup path:
`https://vitest-evals.sentry.dev/docs`
- Use [packages/vitest-evals/README.md](packages/vitest-evals/README.md) for
the core package authoring model and examples.
- Use [docs/github-actions.md](docs/github-actions.md) when changing CI
reporting behavior.
- Use [docs/architecture.md](docs/architecture.md) and
[docs/development-guide.md](docs/development-guide.md) before changing package
boundaries or product shape.
- Follow [policies/docs-writing.md](policies/docs-writing.md) for docs copy,
hierarchy, and visual consistency.
## Workspace Layout
```text
packages/
vitest-evals/
core/
harness-ai-sdk/
harness-openai-agents/
harness-pi-ai/
github-reporter/
report-ui/
apps/
demo-ai-sdk/
demo-openai-agents/
demo-pi/
```
## Development
```sh
pnpm install
pnpm lint
pnpm typecheck
pnpm test
pnpm build
pnpm evals
pnpm evals -- --info
pnpm evals:info
pnpm evals:verbose
```
Eval output has two report levels:
- normal: compact one-line eval and harness summaries
- info: per-tool summaries, arguments, timing/size metadata, replay status, and
final output summaries
`--verbose` and `-v` are accepted as aliases for `--info` for compatibility.
Full transcripts and spans stay in the JSON report metadata.
The root Vitest config is intentionally small. Package name resolution comes
from the workspace `tsconfig` paths via `vite-tsconfig-paths`, and package
boundaries are expressed in package manifests rather than hard-coded alias
tables.
Pull request CI runs the same core safety checks: release config validation,
lint, typecheck, the CI test suite, and the workspace build.
## Docs Site
The public docs site lives in `packages/docs` and is built with Astro for
Vercel. It is configured for `https://vitest-evals.sentry.dev`.
```sh
pnpm run docs
pnpm run docs:build
```
## GitHub Reporting
For GitHub Actions, emit Vitest JSON and let the GitHub reporter read that file.
The JSON report preserves `task.meta`, including eval scores, harness runs,
usage, and tool calls. JUnit can still be emitted alongside it for CI systems
that expect XML.
```yaml
steps:
- name: Run evals
run: |
pnpm exec vitest run apps packages \
--config=./vitest.config.ts \
--reporter=vitest-evals/reporter \
--reporter=json \
--outputFile.json=vitest-results.json
- uses: getsentry/vitest-evals@v0
if: always()
with:
results: vitest-results.json
publish-check: true
min-pass-rate: 0.8
```
The reporter action writes a plain ASCII job summary through
`GITHUB_STEP_SUMMARY`, emits terse annotations for failed evals, and can publish
a separate Check Run when `publish-check` is enabled and `checks: write`
permission is configured. On `pull_request`, that Check Run attaches to the PR
head commit (not the temporary merge SHA). It can also reduce sharded eval JSON
artifacts into one combined report and gate CI on pass rate or average score.
See [docs/github-actions.md](docs/github-actions.md) for the minimal workflow,
soft-fail defaults, and score-gate options.
## Local Report UI
Use `vitest-evals serve` to inspect one or more JSON reports locally. It
accepts files, simple globs, and directories of JSON artifacts, then serves a
React SPA with run summaries, eval cases, separate application and judge usage,
harness output, sessions, tools, and trace details.
```sh
pnpm exec vitest-evals serve vitest-results.json
pnpm exec vitest-evals serve "eval-results/*.json"
pnpm exec vitest-evals serve eval-results/
```
## Releases
Run the manual Release workflow. Leave `bump` at `auto` to pick the bump from
the commits since the last stable release, or select `patch`, `minor`, or
`major`. Before `1.0.0`, `auto` turns breaking changes into a minor bump.
For a preview release that should not become the main stable version, select
`pre`. It cuts an `alpha` prerelease of the version `auto` would release: from
`0.8.0`, a minor bump produces `0.9.0-alpha.0`, and running `pre` again
produces `0.9.0-alpha.1`. For a `beta` or `rc` line, set `version` to the exact
version, such as `0.9.0-beta.0`. `version` overrides `bump`. Craft publishes
npm prereleases under a prerelease dist-tag so the stable `latest` line is not
moved.
The root GitHub Action runs the bundled `github-reporter/dist/action/index.js`,
which is committed on `main`. On pull requests from this repository, CI
rebuilds the bundle and commits it back to the branch when it changed. The
push runs in a separate `workflow_run` workflow from `main`, so pull request
runs never get the release bot key. That workflow runs only from the default
branch, so a change to it takes effect after it merges. On fork
pull requests and on `main`, CI cannot push, so it fails when the bundle is
stale. Then run `pnpm build:action` and commit the result. CI also checks the
bundle on the merge with `main`. If that check fails after the branch is fresh,
merge or rebase on `main` so CI rebuilds the bundle for the merged code.
Craft tags the release commit as `vX.Y.Z`, so workflows can use
`getsentry/vitest-evals@vX.Y.Z`. On stable releases, Craft also moves the
floating `vX` tag (for example, `v0`) to the same commit. Prereleases do not
move it. Craft ignores `vX` when it looks up the previous release, so `auto`
still starts from the latest `vX.Y.Z`. The Craft GitHub target creates the
release but is filtered away from package artifacts so it does not upload npm
package contents as release assets.
Releases up to 0.18.0 tagged the bundle on a commit outside `main`, so `main`
only reaches their `vX.Y.Z-src` tags. Craft would use `v0.18.0-src` as the
previous release: `auto` would compute `0.18.0`, which already exists, and
`major` or `pre` would cut the wrong version. A release check stops the
workflow when the latest `vX.Y.Z` tag on `main` does not match the package
version. For the first release after this change, set `version` to the exact
version (for example, `0.18.1`). Then check that
`git ls-remote origin refs/tags/v0` points at the `v0.18.1` commit: Craft only
logs a warning when it cannot move a floating tag. After that, `auto` and the
bump types work.
Release config validation treats every non-private `packages/*/package.json` as
publishable. `pnpm release:check` verifies that `.craft.yml`, the post-merge
pack step, Craft artifact selectors, package version bumping, and the bundled
action commit all cover that manifest-derived package set.
## Example
The `apps/demo-pi` app shows the intended explicit-run flow:
```ts
import { getModel } from "@mariozechner/pi-ai";
import { expect } from "vitest";
import { piAiHarness, piAiJudgeHarness } from "@vitest-evals/harness-pi-ai";
import {
describeEval,
FactualityJudge,
toolCalls,
} from "vitest-evals";
import { createRefundAgent } from "../src/refundAgent";
const judgeHarness = piAiJudgeHarness({
model: getModel("anthropic", "claude-sonnet-4-5"),
temperature: 0,
});
const factualityJudge = FactualityJudge({ judgeHarness });
describeEval(
"demo pi refund agent",
{
harness: piAiHarness({
agent: () => createRefundAgent(),
}),
judges: [factualityJudge],
judgeThreshold: 0.6,
},
(it) => {
it.for([
{
name: "approves refundable invoice",
input: "Refund invoice inv_123",
expected: "The refund request is approved.",
expectedStatus: "approved",
expectedTools: ["lookupInvoice", "createRefund"],
},
])("$name", async ({ input, ...expected }, { run }) => {
const result = await run(input);
expect(result.output).toMatchObject({
status: expected.expectedStatus,
});
expect(toolCalls(result).map((call) => call.name)).toEqual(
expected.expectedTools,
);
await expect(result).toSatisfyJudge(factualityJudge, {
expected: expected.expected,
threshold: 0.6,
});
});
},
);
```
Harness-backed suites stay close to plain Vitest:
- `describeEval(...)` binds a suite-level harness
- tests call `run(...)` explicitly
- ordinary `expect(...)` assertions stay first-class
- judges layer in through `expect(...).toSatisfyJudge(...)`
- every judge is a named object with `assess(ctx)`
- every judge receives `JudgeContext` with typed `input`, typed `output`, the
normalized run/session, and tool calls; `output` is only optional
when the harness output type includes `undefined`
- judges own their prompt, rubric, and parsing; LLM-backed judges use
`ctx.runJudge(...)` from a configured `judgeHarness`
- pass scenario-specific judge criteria explicitly to matcher options, or bind
suite-wide criteria on the judge instance
- reporter output separates application, judge, and combined token and
`costUsd` usage from normalized runs. First-party harnesses attach native
spans when provider/runtime data is available; `createHarness(...)` attaches a fallback
run span when a custom harness does not return traces itself. Span attributes
include typed OpenTelemetry GenAI semantic keys while keeping
provider-specific keys JSON-safe.
Built-in judges include `FactualityJudge()` for model-backed factuality
grading plus deterministic helpers such as `StructuredOutputJudge()` and
`ToolCallJudge()`. Configure the judge-side model separately with
`FactualityJudge({ judgeHarness })`, `describeEval({ judgeHarness })`, or
matcher options only when a judge calls `ctx.runJudge(...)`. A `judgeHarness` is
a dedicated judge-model adapter, so the same factuality judge can be reused
across AI SDK, Pi, OpenAI Agents, or custom app harnesses. Automatic inference
for explicit matcher calls only applies when suite judges share the same judge
harness instance.
Tool replay is available for opt-in tools in the first-party harnesses.
Configure the replay mode and directory globally in Vitest, then opt individual
tools in from the harness with `toolReplay: { toolName: true }`.
```ts
import tsconfigPaths from "vite-tsconfig-paths";
import { defineConfig } from "vitest/config";
export default defineConfig({
plugins: [tsconfigPaths()],
test: {
include: [
"packages/**/*.test.ts",
"packages/**/*.eval.ts",
"apps/**/*.test.ts",
"apps/**/*.eval.ts",
],
env: {
VITEST_EVALS_REPLAY_MODE: "auto",
VITEST_EVALS_REPLAY_DIR: ".vitest-evals/recordings",
},
},
});
```
`auto` (the default) replays when a recording exists and writes a new one
otherwise. `record` always calls live and overwrites recordings — use it to
refresh fixtures intentionally. Recordings are stored under
`.vitest-evals/recordings/<tool-name>/`.
`pnpm evals` fans out to each workspace package or app that exposes an `evals`
script. The shared eval CLI defaults replay to `auto` and writes recordings
under `.vitest-evals/recordings`, unless those environment variables are
already set. Demo apps expect provider keys in `.env` or `.env.local`. The
intentional failing examples remain under the `evals:fail` scripts.