pi-evals

name	pi-evals
description	Write and run evals for pi extensions and agent behavior using @aliou/pi-evals. Use when creating eval files, writing custom scorers, configuring eval runs, or testing that pi extensions work correctly.

pi-evals

Eval framework for testing pi coding agent behavior. Runs prompts against pi via createAgentSession, then scores the results.

Quick Start

Install:

pnpm add -D @aliou/pi-evals

Create pi-evals.config.ts at the project root:

import { defineConfig } from "@aliou/pi-evals";

export default defineConfig({
  defaults: {
    model: "claude-haiku-4-5",
    provider: "anthropic",
  },
  evalsDir: "./evals",
  timeout: 60_000,
});

Create an eval file in evals/:

// evals/hello.eval.ts
import { evaluate, Scorers } from "@aliou/pi-evals";

evaluate("Create hello file", {
  config: {
    model: ,
    : ,
  },
  : [
    {
      : ,
      : { : { :  } },
    },
  ],
  : [.()],
  : ,
});