| name | prompt-evaluation-runner |
| description | Use when evaluating prompts, LLM outputs, red-team suites, or model behavior with local eval configs and safe provider/cost controls. |
Prompt Evaluation Runner
When to use
Use this skill when you need to evaluate an LLM app, test a prompt, or run red-teaming/vulnerability scans against a target model or application.
Requirements / Checks
- Check if an evaluation tool is defined in project deps, scripts, lockfiles, or local toolchain.
- Do not run external commands like
npx Prompt evaluation@latest directly.
- If missing, ask before adding a local dev dependency or using an ephemeral runner.
- Confirm expected cost, provider, API keys, and network target before execution.
Workflow
- Define risk: State target behavior, failure mode, provider(s), and budget limits.
- Choose assertions: Prefer deterministic checks first: exact/contains/regex/JSON/schema/javascript/python/cost/latency.
- Use model graders sparingly: Pin grader provider/model and explain cost/non-determinism.
- Configure minimally: Keep config to description, env refs, prompts, providers, default assertions, and tests.
- Handle env safely: Use templated env references such as
{{env.NAME}}; never hardcode keys.
- Execute locally: Run smallest suite first. Ask before long, paid, red-team, or production-targeted runs.
- Analyze failures: Separate prompt failures, provider variance, flaky graders, bad fixtures, and config mistakes.
Safety Constraints
- Do NOT log, echo, or store secrets (API keys) in configuration files or chat output.
- Do NOT run evaluations against production endpoints without user consent.
- Avoid executing arbitrary remote code or unvetted plugins during evaluation.
Validation / Done Criteria
- Evaluation config is valid, minimal, and uses safe env refs.
- Deterministic assertions exist where possible.
- Run scope, provider, and cost are reported.
- Results are summarized without leaking sensitive data.
References
references/eval-config-patterns.md
