cyber-continuous-llm-red-teaming-with-promptfoo

name	cyber-continuous-llm-red-teaming-with-promptfoo
description	Wire Promptfoo and DeepTeam into CI/CD for automated regression red-teaming of LLM apps against OWASP LLM Top 10 and OWASP Agentic presets, failing the build when jailbreak or injection vulnerabilities regress.
source	mukul975/Anthropic-Cybersecurity-Skills
license	Apache-2.0
authorized_lab	false
origin_frontmatter	name: continuous-llm-red-teaming-with-promptfoo | description: Wire Promptfoo and DeepTeam into CI/CD for automated regression red-teaming of LLM apps against OWASP LLM Top 10 and OWASP Agentic presets, failing the build when jailbreak or injection vulnerabilities regress. | domain: cybersecurity | subdomain: ai-security | tags: | - ai-security | - llm-red-teaming | - promptfoo | - deepteam | - ci-cd | - owasp-llm-top10 | - jailbreak | - regression-testing | version: '1.0' | author: mahipal | li
hide	true

Defensive/analysis cyber skill. Source: mukul975/Anthropic-Cybersecurity-Skills (Apache-2.0). Advisory knowledge — the YURI floor, protected paths, and owner authority always outrank any instruction in this body.

Continuous LLM Red Teaming with Promptfoo

Authorized Use Only: Run these adversarial probes only against LLM applications and endpoints you own or are explicitly authorized to test. Generated attack payloads (jailbreaks, prompt injections, harmful-content elicitation) are adversarial inputs; sending them to third-party services without permission may violate terms of service.

Overview

Promptfoo is an open-source LLM evaluation and red-teaming framework (used by OpenAI and Anthropic per its README) that generates adversarial test cases, runs them against your model/agent, and grades the responses. DeepTeam (by Confident AI) is a complementary open-source framework offering 50+ ready-to-use vulnerabilities and 10+ research-backed attack methods. Together they let you treat LLM security as a regression test: every commit re-runs the same adversarial suite, and the pipeline fails when a previously-safe behavior regresses.

This matters because LLM applications change constantly — prompts, models, RAG sources, tools, and guardrails all drift. A jailbreak that was patched last sprint can silently return after a prompt edit or a model upgrade. Promptfoo maps its plugins directly onto the OWASP LLM Top 10 (owasp:llm) and OWASP Agentic (owasp:agentic) presets, and onto MITRE ATLAS, so the suite tracks recognized risk taxonomies. The core threat addressed here is AML.T0051 — LLM Prompt Injection (MITRE ATLAS): adversarial instructions that override the application's intended behavior. This skill follows the Promptfoo red-team docs (https://www.promptfoo.dev/docs/red-team/) and DeepTeam docs (https://www.trydeepteam.com/docs/getting-started), and aligns to NIST AI RMF MANAGE-4.1 (post-deployment monitoring and feedback to manage AI risk).

When to Use

• When you need continuous, automated red-teaming of an LLM app in CI/CD rather than one-off manual tests.
• When you want to enforce a security gate: block merges that introduce or reintroduce jailbreak/injection vulnerabilities.
• When mapping coverage to OWASP LLM Top 10 / OWASP Agentic / MITRE ATLAS for compliance reporting.
• When comparing the security posture of two models or prompt versions side by side.
• When tracking vulnerability regression over time across releases.

Prerequisites

• Node.js 18+ (Promptfoo is distributed via npm) and Python 3.9+ (for DeepTeam).