# Supabase Evals
This repo answers how well can agents use Supabase across various tasks.
## Quickstart
Clone with submodules:
```bash
git clone --recurse-submodules git@github.com:supabase/evals.git
```
If you already cloned without submodules:
```bash
git submodule update --init
```
From the repo root:
```bash
pnpm install
cp .env.example .env
```
Agent-backed runs require the relevant provider key in `.env` (e.g. `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`)
## Concepts
- An **eval** is one scenario under `evals/<suite>/<id>/`. It contains the prompt, scorer, and optional starting state for the two environments: `remote/` (the hosted project) and `local/` (the agent's working files).
- An **experiment** is one agent/runtime/model setup under `experiments/<suite>/<name>.experiment.ts`. Its ID remains `<name>`.
- A **suite** is a named set of evals and the experiments that run them. Each suite gets a folder under `evals/` and `experiments/`. Experiments used in several suites can use a shared preset from `experiments/presets.ts`. The suites are `benchmark`, `regression`, `docs`, and `cli`.
- An **agent** is the model driver that receives the eval prompt and calls the configured tools.
- A **runtime** is the local Supabase-like environment and tool surface an experiment gives to the agent.
- `platform-lite` exposes a Supabase Management API-compatible HTTP surface backed by [`@supabase/lite`](https://github.com/supabase/supabase-lite), so real tools like `@supabase/mcp-server-supabase` can run against a lightweight project.
## Running evals
Running evals executes experiment x eval pairs and writes local result files under `results/`.
Run a single eval with one experiment:
```bash
pnpm eval -- --eval resolve-dataapi-001-empty-results --experiment claude-code-sonnet-5.5
```
Run selected evals across multiple experiments:
```bash
pnpm eval -- \
--experiment claude-code-sonnet-5.5 \
--experiment claude-code-opus-5.5 \
--eval resolve-dataapi-001-empty-results \
--eval investigate-auth-001-deleted-user-access
```
`--suite`, `--experiment`, and `--eval` accept multiple inputs via repeated flags as well as comma-separated values. `--experiment` takes the experiment name. Each eval runs experiments from its own suite folder.
Use `--runs N` to score each experiment and eval pair independently `N` times.
Each run writes to `results/<suite>/<experiment>/<eval>/run-<n>/result.json`. Add
`--skip-existing` to resume without replacing completed runs:
```bash
pnpm eval -- --eval resolve-dataapi-001-empty-results \
--experiment claude-code-sonnet-5.5 \
--runs 3 \
--skip-existing
```
Run every experiment in `experiments/benchmark/` across all benchmark evals:
```bash
pnpm eval -- --suite benchmark
```
### View results in the web app
Set `BRAINTRUST_API_KEY` and `BRAINTRUST_PROJECT_ID` in the repository's `.env`, then run:
```bash
pnpm web
```
The server fetches results once at startup. See [CONTRIBUTING.md](CONTRIBUTING.md#publishing-and-consuming-results) for publishing and querying results.
## Eval Shape
Every eval contains:
1. `PROMPT.md` - frontmatter metadata plus the task description the agent sees.
2. `EVAL.ts` - a default-exported scorer.
3. Optional `remote/` - the hosted project's starting state, seeded into platform-lite: `project.sql` (database), `logs.jsonl` (observability logs), `functions/` (already-deployed edge functions).
4. Optional `local/` - the agent's starting files, copied into the sandbox workspace the agent works in (absent means an empty workspace, or no sandbox at all for tools evals).
The two directories mirror Supabase's two environments: `remote/` describes what the customer's hosted project already looks like, `local/` describes what the developer's working directory already looks like.
`PROMPT.md` frontmatter drives eval discovery and site filters:
```md
---
stage: build
product:
- database
- auth
topic:
- rls
- security
motivation: AI-123
---
```
Allowed metadata values are defined in `packages/core/src/eval-metadata.ts`.
## Eval Modes
There are two runtimes, chosen automatically per eval:
- **Tools evals** run the agent against the experiment's MCP/tool surface (no `local/` directory, no `interface: cli`), then score the resulting project state or report.
- **Local-stack evals** run the agent inside a Docker sandbox — a `bash` tool plus file tools with the real Supabase CLI installed — so it can run `supabase init/start/db/test` against a real local stack. An eval uses this runtime when it ships a `local/` workspace **or** declares `interface: cli` (the latter covers bootstrap scenarios that start from an empty workspace).
`interface` (`mcp` | `cli`) is otherwise a benchmark dimension (a cross-team KPI label), not the runtime switch — the `local/` directory and `interface: cli` are what decide whether a sandbox boots.
### Local-stack evals
The Supabase CLI is the agent's tool; the **local stack** (the Docker services `supabase start` runs on a developer machine) is the environment it acts on — distinct from the remote/hosted platform that platform-lite mocks. Experiments declare the environment like MCP servers and skills: add `localStack: localStackRuntime()` (from [`@supabase-evals/sandbox`](packages/sandbox/src/local-stack-runtime.ts)); experiments without it skip these evals. Skills compose with the CLI tools as usual, and tool surfaces merge, so an experiment can in principle expose MCP and CLI together.
**Scoring uses host tooling against an exported workspace.** After the agent finishes, the harness copies its workspace out of the sandbox to the host (`docker cp`), so scorers can run the repo-root `vite`/`vitest` against the produced files without that toolchain having to exist in the sandbox — the same build/test scoring former "project" evals used. Scorers may also run commands and SQL **inside** the sandbox (against the live stack) via the scoring context.
Local-stack evals require a running Docker daemon. Each attempt boots a fresh sandbox container that mounts the host Docker socket, so `supabase start` spawns the local stack as sibling containers; the sandbox runs with host networking, so their published ports land directly on the sandbox's `127.0.0.1` default ports. Supabase's default host ports (54321-54329) must be free — stop any local `supabase start` stacks before running.
An eval's optional `local/` directory is copied into the sandbox workspace before the agent starts. A `services:` frontmatter list declares which local-stack services the scenario needs (e.g. `gotrue`, `kong`, `postgrest`); every other service is excluded from `supabase start` — including when the agent runs it itself — to keep stack boots fast. An empty list (`services: []`) starts only the database; omit the key entirely to start the full stack.
Set `cliVersion: 2.109.1` in an eval's frontmatter when it requires a specific Supabase CLI release. This overrides an experiment's `localStackRuntime({ cliVersion })` setting; otherwise the runtime setting or repository-wide default applies.
An experiment can instead pass `localStackRuntime({ cliVersion: 'stable' })`, `'beta'` or `'next'` to track npm's dist-tag for the `supabase` package rather than an exact version, resolved once at session start. An eval's own `cliVersion:` pin still wins over either form.
An experiment can pass `localStackRuntime({ docker: 'no-daemon' })` or `'absent'` to stage a sandbox where the Docker daemon is unreachable or the `docker` binary is missing entirely, instead of the default `'available'`. `needsDocker` defaults to `true`; set it `false` in an eval's frontmatter when the scenario can run, and is meaningful, without a Docker daemon (e.g. starting the stack is the agent's own job) — that's what lets a Docker-less experiment pick the eval up.
An experiment can pass `localStackRuntime({ env: { SUPABASE_EXPERIMENTAL_STACK: '1' } })` to turn on feature-flagged CLI behaviour for every eval it runs. The eval itself stays flag-agnostic, so once the feature graduates you drop the entry and the eval's history carries on. Names the harness manages itself (`PATH`, `HOME`, `DOCKER_HOST`, hosted-link credentials and so on) are rejected.
Scorers check what the agent produced, never what the harness provisioned: with `projectRunning: true` (the default) the running stack and the seeded `local/` workspace are setup, so score only the deltas the agent made on top; with `projectRunning: false` the agent creates that state itself, so depending on it is fair game.
Test the sandbox plumbing without an agent run (Docker required, not part of `pnpm test`):
```bash
pnpm --filter @supabase-evals/sandbox test:docker
```
## Skills
Skills come from [`supabase/agent-skills`](https://github.com/supabase/agent-skills), pinned as a git submodule at `submodules/agent-skills`. The `skills/` directory contains symlinks into the submodule.
To use a skill in an experiment, reference its directory name in the experiment's `skills` array.
Skills are installed into the sandbox workspace with [Vercel's `skills` CLI](https://github.com/vercel-labs/skills) under each harness's native skills folder (`.claude/skills/` for Claude Code, `.agents/skills/` for Codex and OpenCode). The `ai-sdk` harness exposes a `load_skill` tool to support skill loading.
## Framework Checks
```bash
pnpm typecheck
pnpm test
```
Typechecks every package, then runs the local tests.
## Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md) for guidance on adding evals and experiments, and submitting changes.