ai-evals

name	ai-evals
description	Author and run black-box benchmark cases for the Windmill AI generation modes (flow/app/script/cli/global) in ai_evals/. Use when adding or changing eval cases, or when running before/after benchmarks for AI chat / copilot changes.

AI evals — authoring and running benchmark cases

ai_evals/ is a black-box benchmark runner for the Windmill AI generation modes: flow, app, script, cli, global. It always tests the current production prompts, tools, and guidance in this checkout. Each attempt runs the real production path, deterministic validation, then LLM judging.

The goal is to test current production guidance with realistic user requests — not to pin one exact implementation shape.

Running benchmarks

cd ai_evals
bun install                       # first time; frontend modes also need `cd frontend && bun install`
bun run cli -- models             # list model aliases
bun run cli -- cases global       # list cases for a mode
bun run cli -- run global global-test1-script-create --model sonnet

Frontend modes (flow/script/app/global) route model calls through a Windmill backend's /api/w/<ws>/ai/proxy, so you need any reachable backend:

WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1:<port> WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests \
  bun run cli -- run global <caseIds...> --models sonnet,gpt-5.5,gemini-3.1-pro-preview

• Reuse an existing workspace. CE builds cap workspaces, so temp-workspace creation 400s ("reached workspace limit"). Always set (or any existing workspace) to reuse one. The only side effect of a run is upserting an resource there.