| name | promptfoo |
| description | Context-efficient reference for building, configuring, and optimizing promptfoo evals, providers, and assertions locally. Use this skill before writing or debugging promptfoo-related configs and custom scripts. |
Promptfoo Knowledge Skill
Use this skill when the task involves creating, editing, or debugging promptfoo configuration (promptfooconfig.yaml), providers, custom evaluation scripts, or CLI workflows.
When to Run
- Setting up a new promptfoo eval or project
- Configuring providers (built-in, local, or custom)
- Writing or debugging custom JavaScript/TypeScript/Python providers
- Writing or debugging custom assertions or transforms
- Optimizing prompts or comparing models
- Troubleshooting failed evals, provider errors, or OOM/timeouts
- Converting between CLI, YAML, and Node-package usage
What Promptfoo Is
promptfoo is an open-source CLI and library for evaluating and red-teaming LLM applications. It runs prompts against one or more providers (LLM APIs), compares outputs, and scores them with assertions and metrics. The core idea is test-driven prompt engineering: define expectations first, then iterate.
Mental Model
prompts × providers × tests = matrix of outputs
|
v
assertions & metrics
- Prompts: text, chat JSON, or dynamically generated templates
- Providers: LLM targets (OpenAI, Anthropic, local Ollama, custom JS/Python, HTTP, etc.)
- Tests: input variables + assertions that define pass/fail criteria
- Config: a single
promptfooconfig.yaml binds them together
Phase Map
Read phases in order when learning the tool; jump directly to a phase when the task is focused.
| Phase | File | Covers |
|---|
| 0 | SKILL.md | Index, when-to-use, mental model (you are here) |
| 1 | phases/01-overview.md | Philosophy, workflow, key concepts |
| 2 | phases/02-quickstart-workflow.md | Install, init, eval, view, .env |
| 3 | phases/03-config-reference.md | promptfooconfig.yaml structure and properties |
| 4 | phases/04-prompts.md | Text, file, chat, dynamic, Nunjucks |
| 5 | phases/05-providers.md | Built-in providers, syntax, configuration, env vars |
| 6 | phases/06-assertions-metrics.md | Deterministic, model-graded, custom, derived, named |
| 7 | phases/07-custom-providers.md | JS/TS/Python provider interfaces and return contracts |
| 8 | phases/08-evaluation-scripts.md | Custom asserts, transforms, Node package API |
| 9 | phases/09-cli-commands.md | CLI commands, environment variables, CI behavior |
| 10 | phases/10-debugging-scenarios.md | Troubleshooting, logs, OOM, stuck evals, custom-provider debugging |
| QR | quick-reference.md | Compact cheat sheet |
Quick Decision Guide
- New eval? Start at
phases/02-quickstart-workflow.md, then phases/03-config-reference.md.
- Need a custom model/API? Read
phases/07-custom-providers.md.
- Need a custom scorer? Read
phases/08-evaluation-scripts.md.
- Something is broken? Read
phases/10-debugging-scenarios.md first.
- Need a property list? Read
phases/03-config-reference.md or quick-reference.md.
