design-ai-redteam-plan

name

design-ai-redteam-plan

description

Scope an AI red-team engagement by threat-modeling the system (assets, attackers, trust boundaries), splitting safety from security, traversing the attack-taxonomy decision tree to a prioritized OWASP LLM Top 10 / MITRE ATLAS attack list, and setting the rules of engagement plus likelihood×impact success and severity criteria. Reach for this when the user asks "how should we red-team this LLM feature?", "what should we attack first?", "is this a safety or a security problem?", or "what are the rules of engagement?". Used by `ai-redteam-lead` (primary).

Skill: design-ai-redteam-plan

Invoked by: ai-redteam-lead (primary). Also consulted by adversarial-testing-engineer when execution reveals the scope or threat model is wrong.

When to invoke: "How should we red-team ?"; "what should we attack first?"; "safety or security problem?"; "what are the rules of engagement / disclosure?"; any move from a deployed AI system to a scoped adversarial plan.

Output: the threat model + safety-vs-security split + the prioritized attack taxonomy (OWASP LLM Top 10 / MITRE ATLAS) + rules of engagement + likelihood×impact success/severity criteria, captured in the red-team plan.

Procedure

1. Characterize the system under test. Model(s) and version; what it reads (user turns only, or web/docs/RAG/tool results — attacker-controllable?); what it can do (text only, or tools/code/email/payments/browsing); what it knows/reaches (system prompt, user PII, training corpus, other tenants); modalities; and its trust boundaries (where untrusted input crosses into a privileged context). A plan with no system characterization is guessing.
2. Threat-model: assets × attackers × abuse cases. Name the assets (system prompt, RAG data, downstream actions, user data), the attackers (external user, a poisoned document, a compromised dependency), and the abuse cases (what unauthorized outcome each attacker wants). This is the source the taxonomy is derived from — not a blog's jailbreak list.
3. Split safety from security. For each concern, label it: security (an adversary induces a boundary violation — this team) vs safety (the model produces harmful content on its own — often trust-and-safety). Route the safety half out; keep the security half.
4. Traverse the attack-taxonomy decision tree in ../../knowledge/ai-attack-taxonomy-decision-tree.md against those inputs:
   • reads attacker-controllable content → indirect prompt injection (LLM01) leads,
   • user-only input → direct injection / jailbreak (LLM01),
   • can act via tools → excessive agency (LLM06) — high impact,
   • reaches sensitive data / a training corpus → sensitive-info disclosure (LLM02) / extraction / system-prompt leakage (LLM07),
   • accepts images/audio/files → multimodal; RAG/embeddings → vector weaknesses (LLM08),
   • third-party model/data/plugins → supply chain (LLM03) / poisoning (LLM04),
   • always also weigh improper output handling (LLM05), misinformation (LLM09), unbounded consumption (LLM10).
5. Prioritize by likelihood × impact. Rank the in-scope classes against this system's reachability and blast radius (a public unauthenticated bot's system-prompt leak outranks an insider-only exotic). Produce a P0/P1/P2 ordering, not a flat list.
6. Set the rules of engagement. Scope (endpoints/models/versions), in/out-of-bounds targets, data-handling (no real PII to test infra; canary secrets), a kill-switch, rate/impact limits, and the responsible-disclosure terms.
7. Define success + severity criteria and the flip conditions. What counts as a successful attack (reproducible bypass, confirmed leak, unauthorized tool call), the severity scale (likelihood × impact), and the 1-2 facts that would re-prioritize the taxonomy. Capture it all in ../../templates/ai-redteam-plan.md.

Worked example

User: "We're launching a customer-support RAG assistant that browses our help center and can issue refunds via a tool. How should we red-team it?"

• System: RAG assistant, reads the help center (attacker could plant content) + retrieved chunks; can act (refund tool = moves money); knows the system prompt + per-customer data.
• Threat model: assets = the refund action + customer data + system prompt; attackers = a malicious customer, a poisoned help-center/RAG doc; abuse case = "trick it into issuing an unauthorized refund" and "exfiltrate another customer's data".
• Safety-vs-security: "the bot is rude" = safety (→ trust-and-safety); "an injected doc makes it refund attacker" = security (this team).
• Prioritized taxonomy (decision tree): P0 indirect prompt injection (LLM01) → excessive agency (LLM06) via the refund tool (unauthorized refund); P0 cross-customer sensitive-info disclosure (LLM02); P1 system-prompt leakage (LLM07); P1 vector/embedding weaknesses (LLM08) via poisoned chunks; P2 unbounded consumption (LLM10).
• RoE: test on staging with a sandboxed refund tool (no real money), canary customer records, kill-switch, coordinated disclosure.
• Success/severity: a reproducible unauthorized refund = P0 (high likelihood × high impact). Flip condition: if the refund tool gains no dollar cap or human-in-loop, tool-abuse severity rises further.

Guardrails

• Never name attacks before the threat model — the taxonomy is derived from assets × attackers × trust boundaries.
• Always split safety from security and route the safety half to trust-and-safety; this team owns the security half.
• Prioritize by likelihood × impact, not by how novel/interesting the attack is.
• No adversarial testing without rules of engagement — scope, data-handling, kill-switch, and disclosure come first.
• Quality/capability regression eval is not red-teaming → route to llm-evaluation-engineering; non-AI pentest → security-engineering.
• Volatile facts (OWASP edition/IDs, ATLAS technique IDs, jailbreak techniques, model-version behavior) carry a retrieval date and are re-verified before a client commitment. See ../../knowledge/ai-red-teaming-patterns-2026.md.