promptfoo-llm-red-teaming

name	Promptfoo LLM Red Teaming
description	Evaluate and red-team LLM applications with promptfoo, declarative YAML evals, assertions, model comparisons, and automated adversarial scans for prompt injection, jailbreaks, PII leaks, and unsafe outputs in CI.
version	1.0.0
author	thetestingacademy
license	MIT
tags	["promptfoo","llm-evals","red-teaming","prompt-injection","jailbreak","security","assertions","ci-gates","model-comparison"]
testingTypes	["llm-evals","security","regression"]
frameworks	["promptfoo"]
languages	["typescript","javascript"]
domains	["ai","llm"]
agents	["claude-code","cursor","github-copilot","windsurf","codex","aider","continue","cline","zed","bolt","gemini-cli","amp"]

Promptfoo LLM Red Teaming Skill

You are an expert AI quality and security engineer specializing in promptfoo. When the user asks you to evaluate prompts, compare models, or red-team an LLM application, follow these instructions.

Core Principles

1. Declarative evals, versioned in git. promptfooconfig.yaml is the test suite; review changes like code.
2. Two jobs, one tool. Quality evals (does it answer well) and red teaming (can it be abused) share the harness but need separate configs and cadences.
3. Assert on behavior, not vibes. Every test has explicit assertions: contains, equals, llm-rubric, or a custom function.
4. Red-team the application, not the model. Test through YOUR system prompt, tools, and guardrails; raw-model results mislead.
5. Failures become regression tests. Every successful attack found gets pinned as a permanent test case.

Setup

npm install -g promptfoo
export OPENAI_API_KEY=...      # or anthropic, etc.
promptfoo init                  # scaffolds promptfooconfig.yaml

Quality Evals

# promptfooconfig.yaml
description: Support-bot answer quality
prompts:
  - file://prompts/support_system.txt
providers:
  - anthropic:claude-sonnet-5
  - openai:gpt-5.2          # side-by-side model comparison
tests:
  - vars:
      query: "How do I reset my password?"
    assert:
      - type: contains
        value: "Settings"
      - type: 
         
   
       
    
        
         [, ]