| name | promptfoo-evals |
| description | Use when authoring, expanding, validating, running, or triaging Promptfoo eval suites, deterministic assertions, provider-backed semantic tests, red-team plans, or agent/control-plane prompt regression coverage. |
Promptfoo Evals
Purpose
Use this skill to turn Library Ops control-plane and product-agent behavior into repeatable Promptfoo suites.
Read references/lane-guide.md for the lane breakdown, assertion preferences, and
reporting conventions that accompany this skill.
When evaluation work needs live browser automation or reproducible UI-flow
debugging outside Playwright test files, use the installed $playwright skill
alongside this one.
Procedure
- Find existing Promptfoo config, prompts, tests, and result notes.
- Map the behavior under test to one of the repo lanes: config, deterministic smoke, provider-backed semantic, red-team, or release regression.
- Prefer deterministic assertions such as
contains, not-contains, is-json, javascript, or python before model-graded rubrics.
- Use Promptfoo Nunjucks environment syntax such as
{{env.OPENAI_API_KEY}}; do not use shell $OPENAI_API_KEY syntax inside YAML.
- Keep reusable prompts and larger test cases in files under
evals/, then reference them from config.
- Validate every config before running it.
- Record command, provider, model, exit code, and result location in the validation transcript or dogfooding report.
Deterministic control-plane smoke
npm run eval:validate
npm run eval:smoke
Use this lane for source-of-truth wording, artifact taxonomy, no-secret policies, Task Master local-only config, and Codex auth handling.
Use scripts/codex-runtime-env.sh before promptfoo commands so PROMPTFOO_CONFIG_DIR,
PROMPTFOO_LOG_DIR, and promptfoo state/logs stay under TMPDIR rather than
~/.promptfoo.
Provider-backed semantic lane
Use this lane only when a local provider has been approved and configured. Keep provider secrets in local environment or user-level tool config.
OLLAMA_BASE_URL=http://127.0.0.1:11434 scripts/codex-runtime-env.sh npx --yes promptfoo@0.121.15 eval -j 1 -c evals/release/<suite>.yaml --no-cache --output reports/validation/promptfoo-<suite>.json
For the current Library Ops local-provider lane, the validated shape is:
ollama:chat:qwen3.5:0.8b- serial execution with
-j 1
- explicit
OLLAMA_BASE_URL=http://127.0.0.1:11434
Red-team lane
Map the app boundary first. Do not run a broad red-team scan until the target, inputs, auth posture, and unsafe-action policy are explicit.
npx --yes promptfoo@0.121.15 redteam init
npx --yes promptfoo@0.121.15 redteam run --tag git.sha="$(git rev-parse HEAD)"
npx --yes promptfoo@0.121.15 redteam report
Result block
Promptfoo result
Lane:
Config:
Provider/model:
Command:
Exit code:
Assertions added/changed:
Failures:
Security implications:
Next rerun:
