continuous-llm-red-teaming-with-promptfoo

name	continuous-llm-red-teaming-with-promptfoo
description	Wires Promptfoo and DeepTeam into CI/CD for automated, repeatable red-teaming of LLM apps against OWASP LLM Top 10, OWASP Agentic, and MITRE ATLAS presets, failing the build when jailbreak or injection vulnerabilities regress. Use for continuous adversarial testing in CI/CD, a merge-blocking security gate, or comparing model/prompt versions for compliance reporting.
domain	cybersecurity
subdomain	ai-security
tags	["ai-security","llm-red-teaming","promptfoo","deepteam","ci-cd","owasp-llm-top10","jailbreak","regression-testing"]
version	1.0
author	mahipal
license	Apache-2.0
nist_ai_rmf	["MANAGE-4.1"]
atlas_techniques	["AML.T0051"]

Continuous LLM Red Teaming with Promptfoo

Authorized Use Only: Run these adversarial probes only against LLM applications and endpoints you own or are explicitly authorized to test. Generated attack payloads (jailbreaks, prompt injections, harmful-content elicitation) are adversarial inputs; sending them to third-party services without permission may violate terms of service.

Overview

Promptfoo is an open-source LLM evaluation and red-teaming framework (used by OpenAI and Anthropic per its README) that generates adversarial test cases, runs them against your model/agent, and grades the responses. DeepTeam (by Confident AI) is a complementary open-source framework offering 50+ ready-to-use vulnerabilities and 10+ research-backed attack methods. Together they let you treat LLM security as a regression test: every commit re-runs the same adversarial suite, and the pipeline fails when a previously-safe behavior regresses.

This matters because LLM applications change constantly — prompts, models, RAG sources, tools, and guardrails all drift. A jailbreak that was patched last sprint can silently return after a prompt edit or a model upgrade. Promptfoo maps its plugins directly onto the OWASP LLM Top 10 (owasp:llm) and OWASP Agentic (owasp:agentic) presets, and onto MITRE ATLAS, so the suite tracks recognized risk taxonomies. The core threat addressed here is AML.T0051 — LLM Prompt Injection (MITRE ATLAS): adversarial instructions that override the application's intended behavior. This skill follows the Promptfoo red-team docs (https://www.promptfoo.dev/docs/red-team/) and DeepTeam docs (https://www.trydeepteam.com/docs/getting-started), and aligns to NIST AI RMF MANAGE-4.1 (post-deployment monitoring and feedback to manage AI risk).

When to Use

• When you need continuous, automated red-teaming of an LLM app in CI/CD rather than one-off manual tests.
• When you want to enforce a security gate: block merges that introduce or reintroduce jailbreak/injection vulnerabilities.
• When mapping coverage to OWASP LLM Top 10 / OWASP Agentic / MITRE ATLAS for compliance reporting.
• When comparing the security posture of two models or prompt versions side by side.
• When tracking vulnerability regression over time across releases.

Prerequisites

• Node.js 18+ (Promptfoo is distributed via npm) and Python 3.9+ (for DeepTeam).
• Install Promptfoo and DeepTeam:

  npm install -g promptfoo            # or: npx promptfoo@latest
  pip install -U deepteam