# Convex Coding Evals
Convex is an open-source, reactive database that's the best platform for full-stack AI coding.
We ensure that Convex performs well with a large set of models by continuously running evals. Each eval has a set prompts for coding a Convex backend, a set of human-curated solutions, and a script for evaluating the LLM's output. These evals are split up into eight different categories:
- Fundamentals
- Data Modeling
- Queries
- Mutations
- Actions
- Idioms
- Clients
- Components
The most up to date eval runs can be found on our [website](https://convex.dev/llm-leaderboard).
Detailed results from production runs can be visualized at [convex-evals.netlify.app](https://convex-evals.netlify.app/):
We use these evals to tune our [Convex Guidelines](https://docs.convex.dev/ai/), which greatly improve model performance writing Convex code and decrease hallucinations.
## Decision models
The multiple-choice format measures specific Convex knowledge with four plausible
options per question. It has its own benchmark version, linked to the coding
benchmark, with a separate score and leaderboard. A correct choice does not establish that a model can write
or debug a working Convex application.
The accepted bank contains 108 questions covering 90 of the 112 coding evals.
[decision-bank.json](decision-bank.json) records the included questions and the
22 intentional omissions. Questions are averaged within each source eval, then
source evals receive equal weight. Invalid responses and provider failures count
as incorrect; interrupted and filtered runs do not enter the leaderboard.
Three matched option permutations are used by the hosted workflow. Unknown
provider cost remains unknown, including when a retry has missing usage.
Run a small local trial with `OPENROUTER_API_KEY` in your
environment (or pass `--env-file` to read only the chosen provider's key):
```sh
bun run decisions validate
bun run decisions dry-run --provider openrouter --model typesafe/jev-1.13 --limit-evals 2
bun run decisions run --provider openrouter --model typesafe/jev-1.13 --limit-evals 2
```
The local commands write reports and raw evidence to `output-decisions/` without
reporting to Convex. For a full bank, use `--limit-evals 10000 --repetitions 3
--max-requests 648` and choose an explicit `--max-known-cost-usd` budget. Hosted
runs use the manual **Decision Model Evaluations** workflow on `main`, after the
decision benchmark has been minted. Minting does not initiate model calls.
Hosted question/source evidence remains JSON. Final run evidence is a `.json.gz`
file whose digest covers the compressed bytes; repeated journal requests are
represented by hashes checked against the reconstructed requests. The original
local request/attempt logs remain unchanged.
Jev uses OpenRouter's `/api/alpha/decisions` endpoint with native typed questions;
the language models use `/api/v1/chat/completions`. The hosted Jev configuration
pins `typesafe/jev-1.13`, records the returned model ID, probabilities, confidence,
and provider-reported cost, and sends no chat reasoning or output-token settings.
Every model, Jev included, goes through OpenRouter with the existing
OpenRouter credential. There is no direct provider.
Read the [question authoring standard](docs/decision-question-authoring.md) and
[verification guide](docs/decision-verification.md) before changing a question.
All 108 questions have archived executable evidence and hash checks. Five have
supported clean-checkout replay commands; the other 103 still require portability
work. The release asset `decision-verification.tgz` contains the same evidence,
question files, dependency locks and replay scripts.
The [September 23 bank update](docs/decision-bank-2026-09-23.md) describes the four
reviewed changes, local repeated-run results, and decision-only promotion steps.
The shared protocol excludes old local backend database/log files from source
hashing, alongside dependency/generated directories. Those files are runtime
outputs, and are not included in the published source snapshot.
## Running the evaluations
First, install dependencies:
```bash
npm install -g bun
bun install
echo "OPENROUTER_API_KEY=<your OPENROUTER_API_KEY>" > .env
# Only needed for no_guidelines_with_web runs:
echo "EXA_API_KEY=<your EXA_API_KEY>" >> .env
bun run setup
```
Every model runs through OpenRouter, so `OPENROUTER_API_KEY` is the only
required key. `bun run setup` installs root, `evalScores` and `visualizer`
dependencies, copies `.env` from another worktree when it is missing, and runs
`bun run setup:convex`.
`bun run setup:convex` creates ignored `.env.local` files for the shared Convex
development deployment. Codex worktrees run this automatically via
`.codex/environments/environment.toml`; run it manually in existing worktrees or
non-Codex clones before using `evalScores` codegen or the visualizer. `bun run
dev` also runs it automatically.
### Using the CLI (recommended)
The easiest way to run evals is with the interactive CLI:
```bash
bun run evals
```
[](docs/assets/cli.mp4)
This launches an interactive menu where you can:
- Run all evals
- Select specific categories to run
- Select individual evals
- Choose which model(s) to use
#### CLI Commands
| Command | Description |
| ---------------------- | ------------------------------------ |
| `bun run evals` | Interactive mode |
| `bun run evals list` | List all available evals by category |
| `bun run evals models` | List available models |
#### CLI Options
Run evals directly without interactive mode. Always pass a model: without `-m` or `MODELS`, a run uses every curated model in `ALL_MODELS` (about $87 per full pass as of 2026-09-29).
```bash
# Run specific categories
bun run evals run -m anthropic/claude-sonnet-5 -c 000-fundamentals 002-queries
# Run with a specific model
bun run evals run -m anthropic/claude-sonnet-5 -c 005-idioms
# Run with multiple models
bun run evals run -m anthropic/claude-sonnet-5 -m openai/gpt-5.5 -f "000-fundamentals"
# Filter by regex pattern
bun run evals run -m anthropic/claude-sonnet-5 -f "pagination"
```
### Running directly
You can run the eval runner directly:
```bash
MODELS=anthropic/claude-sonnet-5 bun run runner/index.ts
```
You can specify a test filter regex via an environment variable:
```bash
MODELS=anthropic/claude-sonnet-5 TEST_FILTER='data_modeling' bun run runner/index.ts
```
The test will also print out what temporary directory it's using for storing the generated files. You can override this
with the `OUTPUT_TEMPDIR` environment variable.
```bash
MODELS=anthropic/claude-sonnet-5 OUTPUT_TEMPDIR=/tmp/convex-codegen-evals bun run runner/index.ts
```
### Environment variables
`no_guidelines_with_web` gives models client-owned Exa search and page-fetch
tools in our harness. It needs `CLIENT_WEB_TOOLS=1`, `OPENROUTER_API_KEY` and
`EXA_API_KEY`. Local runs also need `DISABLE_CONVEX_REPORTING=1`, and
`bun run evals run -e no_guidelines_with_web -m <model>` sets both flags. See the
[experiment guide](docs/no-guidelines-with-web.md) and
[benchmark source filtering](docs/web-source-filtering.md) for its limits.
| Variable | Description |
| ------------------- | ------------------------------------------------------- |
| `MODELS` | Comma-separated list of models to run |
| `TEST_FILTER` | Regex pattern to filter evals |
| `OUTPUT_TEMPDIR` | Directory for generated output files |
| `CONVEX_EVAL_URL` | Convex deployment URL (e.g. `https://xxx.convex.cloud`) |
| `CONVEX_AUTH_TOKEN` | Auth token for the Convex backend |
### Output
- Per-step progress lines with the eval id
- Per-eval result with pass/fail status and a clickable output dir
## Adding a new evaluation
Note that test or category names cannot contain dashes.
1. Create a new directory under `evals/<category>/<name>/`
2. Add a `TASK.txt` file describing what the LLM should do
3. Add an `answer/` directory with the human-curated solution
4. Add a `grader.test.ts` file with unit tests
5. Run the eval to verify it works
### Implementing the answer
1. Create `schema.ts` first
2. Copy `convex/_generated` from a sibling single-module eval's answer.
`bunx convex codegen` fails without a Convex deployment, and the scorer
regenerates `_generated` when it deploys.
3. Implement solution files
## Writing evals
### What we're testing
Each eval should make one specific claim about a Convex capability. State that claim before writing or changing its task or grader:
- **API knowledge or requested usage:** the task requests a capability or API family, and the eval checks correct implementation. Exact method names can remain unstated when knowing them is part of the claim.
- **Unprompted selection:** the task describes a need, and the eval checks whether the model chooses a particular pattern or component. A miss establishes that the desired choice was not made; it does not establish that an alternative implementation is broken.
- **Behavioral correctness:** the task states the required behavior, and equivalent correct implementations must be accepted unless an implementation constraint is justified by the stated claim.
Be explicit about product requirements, interfaces, edge cases, and relevant scale. Withholding API mechanics can be intentional; leaving product behavior ambiguous is a separate issue. Record what a pass does **not** establish, such as spontaneous API selection, complete pagination, or production-scale correctness.
Review the actual task and context sent to the model, including the experiment, SDK version, and available tools. `no_guidelines` omits the guidelines and research tools, and scoring does not give the model a code-repair loop. Guideline coverage alone is not sufficient evidence of model fault. A wrong answer, an ambiguous task, and a grader defect can coexist.
Before changing a score, separate those three questions: is the requested capability worth measuring, does the prompt support the graded requirement, and does the grader distinguish correct from incorrect implementations? If a prompt change makes an implicit choice explicit, record the changed measurement; do not reinterpret historical scores under the new contract.
### Writing good prompts
1. **Be explicit about schema** - always provide the complete schema in the prompt using TypeScript code blocks
2. **Clear requirements** - for each function, specify:
- Exact function name
- Required arguments and their types
- Expected return type/structure
- Any specific behaviors or edge cases to handle
3. **Scope the context** - describe the feature and its domain constraints. Omit API mechanics only when recalling or selecting them is the intended measurement. Do not assume guidelines are present in every experiment.
4. **Implementation constraints** - specify what files to create, what NOT to do, and any performance considerations that aren't obvious from the guidelines.
### Common pitfalls
1. **Ambiguous requirements** - don't leave function names unspecified; don't use vague terms like "appropriate" without context; always specify exact field names and types
2. **Over-complication** - don't test multiple concepts in one eval; keep schemas focused on the tested concept
3. **Missing context** - describe the problem domain and required behavior clearly; decide separately whether API references belong in this eval's supplied context
4. **Untestable requirements** - make success criteria measurable; specify exact return types; include specific test cases
5. **Over-specification** - do not give away a choice the eval claims to measure. Explicit API instructions are appropriate for usage evals, but change an unprompted-selection eval into a different measurement.
### Grader validation
Use real data and runtime behavior where possible. For required API use, verify that the relevant operation executes or that the returned artifact derives from it. Merely finding an identifier, import, or correctly shaped call is not enough.
Every grader correction needs both sides of a regression matrix:
- Equivalent valid answers: aliases, constants, shorthand and quoted properties, helpers, and module organization permitted by the task.
- Invalid answers that look plausible: unused correct calls, fake local objects, wrong limits, hardcoded defaults, unrelated operations, and incomplete results.
Keep those regression tests separate from scored assertions so adding fixtures does not change an eval's weighting. Validate the reference and targeted alternatives through the real scoring pipeline. A passing reference alone does not validate the grader.
Execution probes also need scrutiny: an incomplete SDK mock can reject valid code. Cover harmless operations allowed by the task, and document sampled paths and unsupported behavior. Neither a source check nor a sampled execution probe proves correctness for every possible program.
The [Astra task-contract review](docs/astra-eval-contract-review.md) applies this process to the recurring failures audited in September 2026.
### Eval structure
Each eval directory contains:
- `TASK.txt` - the prompt sent to the model
- `answer/` - the human-curated reference solution
- `grader.test.ts` - Vitest tests that score the model's output
The default pipeline installs, deploys, typechecks, lints, and tests a backend.
An optional `eval.json` can select another pipeline:
- `{"pipeline":"static"}` runs a grader directly on raw files, for component
selection evals that deliberately tolerate syntax and stale API errors.
- `{"pipeline":"module"}` installs dependencies and runs a grader without a
backend. The grader must typecheck and execute the module against the task's
SDK version. Module prompts follow the task's file list and dependency pins
instead of the default backend scaffolding instructions.
Model output is untrusted, so the scorer installs with
`bun install --ignore-scripts` and gives the install, Convex CLI, tsc, eslint,
local backend and grader processes only an allowlisted environment
(`runner/childEnv.ts`). Answers and graders cannot rely on package lifecycle
scripts or on the runner's API keys and other environment variables. This keeps
credentials out of model code's environment, but it is not a sandbox: that code
still runs as the same OS user as the runner.
### Common eval types
- **Data modeling** - table relationships, index design, schema validation
- **Query patterns** - CRUD, index usage, filtering, joins, pagination, aggregation
- **Actions** - external calls, storage, node runtime, HTTP endpoints
- **Idioms** - internal functions, file organisation, batch patterns, code reuse
## AI grading
`createAIGraderTest` is currently a no-op. Its test body is commented out in
`grader/aiGrader.ts`, so evals that call it get no AI assessment. The rest of
this section describes it as it would run if re-enabled, which would need
`OPENAI_API_KEY`. Graders no longer get the runner's environment, and passing
the key to them would expose it to model code, so re-enabling it means making
the OpenAI call from the runner instead of from inside the grader.
Grader tests can include a lightweight AI-based assessment that reviews the generated project and provides concise reasoning on pass/fail.
The grader builds a prompt from `TASK.txt` plus a manifest of files from the generated output directory and asks a model to decide pass/fail with reasoning. On failure, the reasoning appears directly in the test output and in `run.log`.
### Usage
Add a single standardised test using the helper:
```ts
import { createAIGraderTest } from "../../../grader/aiGrader";
// Basic usage (default name and 60s timeout)
createAIGraderTest(import.meta.url);
// Optional: custom name/timeout
createAIGraderTest(import.meta.url, "AI grader assessment", 60000);
```
## Generating guidelines
```bash
bun run build:release
```
This will generate guideline files in the `dist/` directory for various AI coding assistants.
## Listing models
```bash
bun run list:models
bun run scripts/listModels.ts --format json
bun run scripts/listModels.ts --due-only --format json
```
## Automated eval workflows
The repo has one scheduled periodic eval workflow:
- `periodic_evals.yml` runs every 4 hours
- each run unions candidates from curated models and up to 15 top-weekly non-curated OpenRouter models
- the combined candidate list is deduped before the workflow matrix expands
The periodic workflow uses the same scheduling policy before it actually queues a model:
- if we have never run a model before, it is due immediately
- otherwise we look at the model's stored OpenRouter first-seen timestamp
- the target interval starts at `24h`, grows with model age, hits about `30d` at one year old, and approaches `60d` for very old models
- the due check uses the latest default-experiment attempt across benchmark versions, so minting a benchmark cannot make every model immediately due
- failed attempts preserve the scheduling cooldown, while `no_guidelines` runs do not delay the next default run
The OpenRouter-derived selectors also do a lightweight preflight check so obviously dead models are skipped before entering the matrix.
### Leaderboard benchmark versions
Benchmark versions are minted manually after a meaningful batch of eval work
is complete. The version itself is a deterministic hash derived from the
scoring protocol, system prompt, guidelines, and complete eval suite. Minting
is metadata-only: it does not queue or run any model, and the periodic workflow
continues using the age- and cost-aware schedule above.
The public leaderboard shows only scores from the current version. Older
versions remain available in the archive. Pre-versioning runs are backfilled
into reconstructed versions using the exact sorted planned-eval set and the
date that suite first appeared. A few historical partial runs use the version
active on their run date, but are excluded from aggregates because their
planned eval count does not match that version.
Runs store a Convex document ID into the `benchmarkVersions` table. New suite
hashes that have not been manually minted point to a private `unminted`
sentinel, so the foreign key is always present without publishing a version
automatically. Local runners are also prevented from reporting to the
production Convex deployment.
After explicit approval, mint through the workflow on `main`:
```bash
gh workflow run mint_benchmark.yml --ref main -f kind=coding # or kind=decision
```
`bun run benchmark:mint` fails without `BENCHMARK_KIND` and refuses production
outside that workflow.