promptfoo

name	promptfoo
description	LLM red teaming and security testing — automated vulnerability scanning for AI agents, RAGs, and LLM pipelines. Covers prompt injection, jailbreaks, data leaks, PII exposure, and 50+ vulnerability types.
triggers	["red team","test llm security","llm vulnerability","prompt injection test","jailbreak test","ai security scan","promptfoo","llm eval","test ai safety"]
linksTo	["security-scanner","owasp","test"]

Promptfoo — LLM Security Testing

Automated red teaming for AI agents, RAG pipelines, and LLM-powered apps. Source: https://github.com/promptfoo/promptfoo | https://www.promptfoo.dev/

Quick Start

# Run red team setup (interactive)
npx promptfoo@latest redteam setup

# Run eval against a target
npx promptfoo@latest redteam run

# View results
npx promptfoo@latest redteam report

Core Concepts

Red Teaming: Automatically generates adversarial prompts to probe:

• Prompt injection / jailbreaks
• Data exfiltration / PII leaks
• Harmful content generation
• Business logic violations
• RAG poisoning / context stuffing

Configuration (promptfooconfig.yaml):

targets:
  - id: openai:gpt-4o
    config:
      systemPrompt: "You are a helpful assistant."

redteam:
  purpose: "Customer support chatbot"
  numTests: 50
  plugins:
    - id: harmful:hate
    - id: pii:direct
    - id: prompt-injection
    - id: jailbreak
    - id: harmful:violent-crimes
  strategies:
    - jailbreak
    - prompt-injection

Key Plugins (50+ vulnerability types)

Category	Plugin IDs
Harmful content	harmful:hate, harmful:violent-crimes, harmful:cybercrime
PII	pii:direct, pii:session, pii:api-db
Injection	prompt-injection, indirect-prompt-injection
Jailbreaks	jailbreak, jailbreak:tree
Business	policy, overreliance, excessive-agency
RAG-specific	rag-poisoning, context-length-exceeded

Integration with UltraThink

Test an agent endpoint:

npx promptfoo@latest eval --config promptfooconfig.yaml

CI/CD integration (GitHub Actions):

- name: LLM Security Scan
  run: npx promptfoo@latest redteam run --ci
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Scan the UltraThink memory API:

targets:
  - id: http
    config:
      url: http://localhost:3333/api/memory
      method: POST
      body: '{"query": "{{prompt}}"}'

When to Use

• Before shipping any LLM feature to production
• After major prompt/system changes
• As part of CI/CD for AI-powered endpoints
• To audit RAG pipelines for data leakage
• When adding new tool use / function calling