pentest-ai-llm

name

pentest-ai-llm

description

LLM-backed application, AI agent, and AI/ML supply-chain penetration testing. Use when the user mentions LLM pentest, AI red team, prompt injection (direct or indirect), jailbreak, DAN, Crescendo, GCG, many-shot jailbreak, system prompt leakage, RAG poisoning, embedding poisoning/inversion, vector DB attack, MCP / Model Context Protocol abuse, tool poisoning, agent excessive agency, function-call abuse, output handling chains (XSS/SSRF/SQLi via LLM output), training-data extraction, membership inference, modelscan/picklescan, Hugging Face poisoned model, garak, PyRIT, promptmap, Augustus, Spikee, OWASP LLM01-LLM10, MITRE ATLAS, or unbounded consumption / cost amplification.

Pentest — AI / LLM

Apply 11-ai-llm/ from the pentest-templates library.

Library root

$PENTEST_LIBRARY — caminho absoluto deste repositório clonado (definido pelo instalador; veja INSTALL.md).

Read 11-ai-llm/00-overview.md first.

Phase order

1. 11-ai-llm/01-information-gathering.md — model + provider + system-prompt indicators + tool inventory + RAG sources + guardrails.
2. 11-ai-llm/04-system-prompt-leakage.md — extract base instructions before attacking; reveals boundaries.
3. 11-ai-llm/02-prompt-injection.md — DAN / encoding / Crescendo / GCG.
4. 11-ai-llm/03-indirect-prompt-injection.md — RAG / web / docs / MCP.
5. 11-ai-llm/07-excessive-agency.md — tool / function-call / MCP abuse.
6. 11-ai-llm/05-rag-and-embeddings.md — vector DB attacks, embedding poisoning, inversion.
7. 11-ai-llm/06-output-handling.md — chain LLM output to web vulns (XSS / SSRF / SQLi).
8. 11-ai-llm/08-sensitive-info-disclosure.md — training-data extraction, PII inference.
9. 11-ai-llm/10-unbounded-consumption.md — DoS / cost amplification.
10. 11-ai-llm/09-supply-chain.md — model registry, dependency CVEs.

How to operate

1. Map the application surface first: model, system prompt indicators, tools/functions, RAG sources, guardrails, output sinks (01-information-gathering.md).
2. Extract the system prompt before anything else (04-system-prompt-leakage.md) — it reveals boundaries and tool definitions, sharply improving every later attack.
3. Quantify success rate (e.g., 7/10 trials) — most bypasses are non-deterministic. Findings without rate quantification are weak.
4. Output-handling chains: when the LLM output reaches an HTML / DB / shell / fetch sink, verify end-to-end (LLM → consumer system → impact). The web-side wraps pentest-web-api.
5. For agent / MCP / tool abuse, document the invocation chain + business impact (the deliverable is the chain, not the individual prompt).
6. Cost amplification: track $/hour budget impact and stop early — token-budget limits are mandatory.

When NOT to use

• Pure web/API testing without LLM features → pentest-web-api.
• Cloud-only LLM endpoints (Azure OpenAI, AWS Bedrock, Vertex) → also pentest-cloud for cloud-side config.
• Mobile app with on-device LLM → also pentest-mobile.

Pre-conditions

• Test account / API key.
• ROE: provider acceptable-use rules, customer-data handling, cost-cap.
• garak (NVIDIA) + PyRIT (Microsoft) installed; promptmap, Augustus, Spikee, modelscan, picklescan, Vec2Text, interactsh available.
• Burp / Caido on HTTP wrapper.

OPSEC

• Provider-side observability is significant — Azure OpenAI / AWS Bedrock log every prompt.
• Aggressive testing → rate-limit / suspension; pace accordingly.
• Production PII may surface in prompts — handle carefully.

Output expectations

• Per-finding: prompt + response (sanitised), reproduction count, success rate, OWASP LLM Top 10 mapping.
• For agent / tool abuse findings: invocation chain + business impact.
• For RAG findings: source document + retrieval evidence.
• For output-handling: end-to-end chain.
• Cost amplification: $/hour budget impact when applicable.

References inside the library

• OWASP LLM Top 10 (2025): https://genai.owasp.org/llm-top-10/
• MITRE ATLAS: https://atlas.mitre.org/
• garak: https://github.com/NVIDIA/garak
• PyRIT: https://github.com/Azure/PyRIT
• Tooling: tool-mappings/01-web-api-tools.md § LLM/AI section.
• Cross-references: 01-web-api/ (HTTP wrapper), 06-cloud/ (cloud-side LLM endpoints), 02-mobile/ (mobile-app LLM features).
• Playbook: playbooks/08-ai-llm-engagement.md.