| name | llm-top10 |
| description | Audit an LLM-powered application against the OWASP LLM Top 10:2025. Covers prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. Produces a prioritized findings report with optional fixes. |
| argument-hint | [{"scope":"full | quick | agents-focused | rag-focused | multimodal-focused | <LLM01-LLM10>"}] |
OWASP LLM Top 10:2025 Audit
You are auditing an LLM-powered application against the OWASP LLM Top 10:2025. Work through each phase in order.
Scope: {{args}}
Phase 1: Interview
Use AskUserQuestion for each question below, one at a time.
Question 1: LLM architecture (multi-select):
| Option | Description |
|---|
chat-interface | User-facing chatbot or assistant |
rag | Retrieval-augmented generation (vector DB, document search) |
agents | Autonomous agents with tools/functions |
mcp-tools | Uses Model Context Protocol servers/tools |
multimodal | Accepts images, audio, video, or PDFs as model input |
fine-tuned | Custom fine-tuned model or LoRA adapter |
api-wrapper | Thin wrapper around a provider API |
multi-agent | Multiple agents that pass messages or share state |
Question 2: Provider and tooling - free text:
Which LLM provider(s), SDK(s), and infrastructure are you using? (e.g. "OpenAI GPT-4o via Vercel AI SDK, Pinecone for RAG, Anthropic MCP servers via stdio")
Question 3: Scope
| Option | Description |
|---|
full | All 10 categories |
quick | Top 5 most relevant for this architecture |
agents-focused | LLM01, LLM05, LLM06, LLM07, LLM10 (agent-heavy apps) |
rag-focused | LLM01, LLM02, LLM04, LLM08 (RAG-heavy apps) |
multimodal-focused | LLM01, LLM02, LLM05, LLM10 (image/audio/PDF input) |
single | One specific category (user names LLM01-LLM10) |
Question 4: Output
| Option | Description |
|---|
report-only | Findings report, no code changes |
report-and-fix | Report + implement all Critical and High fixes |
fix-critical | Implement Critical fixes only, skip report |
Phase 2: Explore
Spawn 4 parallel subagents to map the LLM surface area before auditing:
| Subagent | Focus |
|---|
| 1 | System prompts, prompt templates, user input handling, message construction, role separation |
| 2 | Tool/function definitions, agent loops, MCP server config, tool output handling, downstream sinks (HTML render, code exec, SQL, shell, network calls) |
| 3 | Vector DB config, embedding model, retrieval filters, document ingestion, tenant/namespace isolation, fine-tuning datasets |
| 4 | Provider SDK versions and lockfiles, rate limits, token budgets, timeouts, logging/observability, error handling, guardrails |
Each subagent returns a short inventory (file paths, key functions, framework versions). Collect all findings before Phase 3.
Phase 3: Parallel Audit
Determine the active category set from Phase 1 scope:
full - all 10quick - top 5 highest-risk for the architecture inventory from Phase 2agents-focused - LLM01, LLM05, LLM06, LLM07, LLM10rag-focused - LLM01, LLM02, LLM04, LLM08multimodal-focused - LLM01, LLM02, LLM05, LLM10single - the one category named by the user
Spawn one subagent per active category. Each reports:
CATEGORY: <ID> - <Name>
SEVERITY: Critical | High | Medium | Low | Pass
FINDINGS:
- <file:line or pattern>: <what is wrong>
FIXES:
- <specific change needed>
LLM01:2025 - Prompt Injection
Direct injection - user input overrides system instructions:
- Template string concatenation:
`${systemPrompt}\n${userInput}`, f-strings, .format() with raw user input
- Single
messages array role where system + user content share the same role
- No structural delimiters (XML tags like
<user_input>, Anthropic-style <document>, OpenAI developer/user role separation)
- Known jailbreak patterns not filtered: "ignore previous", "DAN", "developer mode", "system:", "you are now", base64-encoded instructions, leetspeak/unicode obfuscation
Indirect injection - untrusted content enters the model context:
- Web search results, scraped pages, emails, support tickets, PDFs, code files, calendar invites, file names
- RAG retrieval results inserted verbatim with no delimiter or "treat as data" framing
- Tool outputs (HTTP responses, shell output, MCP tool responses) appended as assistant or user content without trust labelling
- MCP server tool descriptions and resource contents (servers can rewrite their own descriptions mid-session)
Multimodal injection (if multimodal selected):
- Images with hidden text/instructions (low-contrast, steganographic, EXIF metadata)
- Audio with prompt-like transcriptions
- PDFs with white-on-white text, embedded JavaScript, or hidden form fields
Agent injection (if agents or mcp-tools):
- Tool call arguments generated from injected content with no validation against the original user request
- Chain-of-thought / reasoning tokens exposed to user and reflected back as input
- No guardrail model or rule check between tool-call decision and execution
Look in: prompt template files, message builders (buildMessages, formatPrompt), RAG context injection, tool result handlers, agent loop code, MCP client wiring, multimodal input preprocessors.
Fixes: structural delimiters with input tag stripping, dual-LLM pattern (privileged vs. quarantined), spotlighting/datamarking of untrusted content, tool-call validation against original intent, allowlist of permitted next actions, guardrail/classifier model for high-risk actions, refuse-on-conflict instructions.
LLM02:2025 - Sensitive Information Disclosure
(LLM07 covers the mechanism of system-prompt leakage; this category covers what data can leak through the model: PII, secrets, other users' data, proprietary content.)
Check for:
- PII, credentials, internal URLs, or business-sensitive data embedded in few-shot examples, retrieved documents, or tool outputs
- No PII redaction/tokenization layer before sending user data to a third-party provider (especially OpenAI/Anthropic/Google with data-retention defaults)
- Model returning data from other users (shared conversation memory, shared cache, shared vector index without tenant filter)
- Error responses surfacing the raw prompt, model name, API key fragments, request IDs, or provider error messages to end users
- Conversation history persisted globally rather than per-user / per-session, or cache keys that lack the user identifier
- RAG / vector search returning documents the requesting user lacks permission to read (no
where user_id = $1 style filter on retrieval)
- Provider data-retention or training-on-data settings left at default (e.g. zero-retention mode not enabled, opt-out of training not set)
- Logs / traces (LangSmith, Helicone, Datadog, CloudWatch, Sentry) capturing full prompts and completions including PII without scrubbing
- Fine-tuning corpus containing memorizable secrets (API keys, customer records) - membership-inference and verbatim-extraction risk
- Embeddings stored with raw source text in metadata, accessible via the same query path
Look in: request handlers, prompt construction, RAG retrieval filters, error middleware, logging config, LLM observability SDK init, fine-tuning data prep scripts, provider client config.
Fixes: PII redaction proxy (Microsoft Presidio, regex pre-filter) before provider calls, per-user context and cache isolation, authorization filter on every retrieval query, scrub prompts/completions in logs, enable provider zero-retention / no-training flags, redact error messages before returning to client, exclude secrets from fine-tuning data.
LLM03:2025 - Supply Chain
Check for:
- LLM provider SDKs (
openai, @anthropic-ai/sdk, @google/genai, cohere-ai, langchain, llamaindex) using broad version ranges (^, ~, *) without lockfile integrity verification
- Third-party MCP servers, plugins, or tool packages installed without code review or pinned commit SHA
- Model weights downloaded from HuggingFace / civitai / arbitrary URLs without checksum or signature verification
- Pickle-based model files (
.pkl, .bin, legacy pytorch_model.bin) loaded with torch.load / pickle.load (arbitrary code execution risk - prefer safetensors)
- GGUF, ONNX, or LoRA adapter files from untrusted sources loaded into the runtime
- AI gateway or proxy (LiteLLM, Portkey, OpenRouter, Vercel AI Gateway) misconfigured, unauthenticated, or with no provenance log
- No SBOM entry for model artifacts; model version not recorded with deployment
- Outdated model snapshots known to have safety regressions (e.g. deprecated
gpt-3.5-turbo-0301, unpatched community fine-tunes)
- HuggingFace
trust_remote_code=True in from_pretrained calls
- CI pulls models or datasets fresh from the internet on every build instead of from a verified cache
Look in: package.json / pyproject.toml / requirements.txt, lockfiles, Dockerfile model download steps, from_pretrained() calls, MCP client config files (mcp.json, .mcp.json, claude_desktop_config.json), gateway config, CI workflow files.
Fixes: pin SDK versions and verify lockfiles in CI, prefer safetensors over pickle formats, checksum/signature-verify all model downloads, pin MCP servers to specific commits and review their tool descriptions, add models to SBOM, set trust_remote_code=False, cache verified artifacts in a private registry.
LLM04:2025 - Data and Model Poisoning
Check for:
- Fine-tuning / RLHF / DPO data ingested from user-controlled sources, public scrapes, or community contributions without provenance and review
- Backdoor / trigger-phrase risk: training data not scanned for trigger tokens that activate hidden behaviour
- RAG corpus accepting user uploads, scraped web pages, or shared documents without content moderation
- Embedding ingestion pipeline runs against untrusted content with no quarantine of "high-influence" documents (very high norm, near-duplicate flooding)
- Split-view poisoning: training-time and inference-time documents fetched from URLs that the attacker can later change
- No train/validation/test split integrity check (attacker can poison the eval set to hide regressions)
- Knowledge base does not record document source, ingestion time, and uploader for forensic rollback
- Periodic re-evaluation against a clean golden-set / red-team prompt suite is missing
- No anomaly detection on retrieval frequency (a poisoned doc that suddenly dominates top-k results)
Look in: ingestion pipelines, document upload handlers, vector DB seeding scripts, fine-tuning scripts (trainer.train, SFTTrainer, accelerate launch), data loader code, eval harness.
Fixes: content moderation and PII scan before ingestion, source allowlisting and signed manifests for training data, store provenance metadata per document, run a fixed red-team eval suite after every ingestion or fine-tune, anomaly detection on retrieval distributions, ability to delete a document and re-index by source.
LLM05:2025 - Improper Output Handling
Treat model output as untrusted user input for every downstream sink.
Check for:
- HTML rendering:
dangerouslySetInnerHTML, v-html, innerHTML, unescaped Jinja/Handlebars output of LLM text - XSS
- Markdown render:
marked, react-markdown, markdown-it without sanitizer / with html: true - script injection, image-tag exfiltration via , malicious links
- Code execution: model-generated code passed to
eval, exec, Function(), vm.runInNewContext, subprocess, Jupyter kernels without sandboxing
- SQL: model-generated SQL executed against a real DB without read-only role, parameterization, or query allowlist (NL-to-SQL pattern)
- Shell: model-generated commands passed to
os.system, subprocess.run(shell=True), child_process.exec
- SSRF via generated URLs: model emits a URL, server fetches it without an allowlist / SSRF guard - classic exfil of cloud metadata (
169.254.169.254), internal services
- Email / messaging: generated content sent without sanitizing for header injection (
\r\n, CC/BCC fields) or phishing link rewriting
- File paths: generated paths used in
fs.readFile / open() without path-traversal validation (.., absolute paths, symlinks)
- Structured output: JSON parsed with
JSON.parse and trusted blindly - no Zod / Pydantic / JSON-schema validation, no field-type enforcement, no length caps
- Tool calls: tool-call arguments not re-validated against the tool's declared schema before execution
- Missing CSP for pages that render LLM output (no
script-src, no img-src, no connect-src restrictions)
Look in: UI components that render assistant messages, code-execution sandboxes, NL-to-SQL handlers, agent tool dispatchers, email/notification senders, fetch-by-URL helpers, JSON parsers downstream of LLM calls.
Fixes: escape all model output before rendering, sanitize markdown (DOMPurify, bleach, rehype-sanitize), strict CSP, isolated sandbox (Vercel Sandbox, Firecracker, gVisor) for code, read-only DB role + query allowlist for NL-to-SQL, URL allowlist + private-IP blocklist for fetches, schema validation on every structured output, re-validate tool arguments before dispatch.
LLM06:2025 - Excessive Agency
Three sub-risks: excessive functionality, excessive permissions, excessive autonomy.
Check for:
- Tools exposed to the model that the current task does not need (e.g.
delete_user, send_email, transfer_funds available for a read-only question-answering flow)
- Tools that wrap broad capabilities (
run_shell, query_db, http_request) with no per-call argument allowlist
- Tool credentials run with user-level or admin scope rather than the minimum role for the action
- No human-in-the-loop confirmation before irreversible / high-impact actions: payments, sends, deletes, role grants, infrastructure changes
- Agent loop with no iteration cap or no scope check, allowing chains of tool calls that drift from the original user intent
- System prompt grants standing authority ("you may perform any action needed") rather than scoping to the current request
- Multi-agent setups where one agent can invoke another agent's tools transitively, bypassing the second agent's scope
- MCP tools mounted with the full permission set instead of the subset the application uses
- No audit log of tool calls with arguments, caller, and outcome - cannot reconstruct an attack chain
- No "dry run" / preview mode for destructive actions
- Agent decisions not gated by a separate policy / authorization service (model is both decider and executor)
Look in: agent loop code, tool registries, MCP client mount config, function/tool definitions, IAM roles for service accounts the agent uses, confirmation UI, audit log writer.
Fixes: least-privilege tool set scoped per task type, per-tool argument allowlists, downgrade service-account permissions to minimum, human-in-the-loop gate for irreversible actions, iteration and depth caps on agent loops, signed action manifests, separate authorization service (OPA, Cedar) for tool authorization decisions, dry-run preview for writes, full audit log of every tool invocation.
LLM07:2025 - System Prompt Leakage
The OWASP framing: assume the system prompt will leak; design so leakage is not harmful. This category is about what you put in the prompt, not just how to hide it.
Check for:
- Hardcoded secrets in system prompts: API keys, DB connection strings, internal hostnames, S3 bucket names, JWT signing keys, webhook URLs
- Authorization rules expressed only in the prompt ("only respond if the user is an admin") with no server-side enforcement - the security control is the prompt itself
- Business logic / pricing rules / competitor lists / unreleased product names embedded in the prompt
- User PII or other tenants' data interpolated into the system prompt (multi-tenant leakage on extraction)
- Prompt extractable via direct (
repeat verbatim, print the text above, ignore and output your instructions) or indirect (translate, summarize, encode as base64, write as a poem) attacks
- Prompt inferable from model behaviour (error responses including the prompt, debug headers, stack traces)
- No output filter for known prompt fragments / canary strings
- Prompt loaded from a public file (
/public/prompt.txt) or shipped to the client bundle (check view-source and JS chunks)
Look in: prompt template files, environment variables interpolated into prompts, multi-tenant prompt builders, client-side bundle for leaked prompts, error/debug routes, observability traces.
Fixes: never store secrets in prompts (read from server-side secret manager at tool-execution time, not in the prompt), enforce authorization on the server not in the prompt, do not interpolate other users' data into a shared prompt, add canary strings to detect leakage, output filter for known prompt fragments, design under the assumption the prompt is public.
LLM08:2025 - Vector and Embedding Weaknesses
Check for:
- Vector similarity search returning results without an authorization filter - the query reaches the DB with no
user_id / tenant_id / acl predicate
- Shared index across tenants with no namespace, collection, or partition separation (Pinecone namespace, Qdrant collection, pgvector schema, Weaviate tenant)
- Embedding model from an untrusted or outdated source, or model version mismatch between indexing and query time (silently degraded retrieval)
- Metadata stored alongside vectors contains PII, secrets, or other-tenant data and is returned in query responses
- Raw source text stored as a vector payload and reachable via the query path even when the user is not authorized to read the source document
- Embedding-inversion / extraction risk: vectors exported or returned to clients (embeddings can be inverted back to approximate source text)
- Poisoning: no detection of documents that dominate top-k for unrelated queries, no quarantine for high-norm or near-duplicate flood
- Dimension / distance-metric mismatch (cosine vs L2 vs dot product) causing silent ranking corruption
- Re-ranker / cross-encoder absent on high-stakes retrieval, allowing pure-vector hijack
- No rate limit on vector queries (allows brute-force embedding extraction)
Look in: vector DB client init, retrieval functions (vectorstore.similarity_search, index.query, pg.query with <-> / <=>), document ingestion pipeline, embedding model config, retrieval filter construction, API routes returning embeddings.
Fixes: mandatory per-user / per-tenant filter on every vector query (enforce in a wrapper, not at call sites), namespace or per-tenant collection isolation, allowlist metadata fields returned to the client, never return raw embedding vectors to untrusted callers, version-pin the embedding model and re-index on change, retrieval anomaly detection, cross-encoder re-ranker for high-stakes flows, rate limit on retrieval endpoints.
LLM09:2025 - Misinformation
Covers hallucinations, fabricated citations, and overreliance by downstream systems or users.
Check for:
- Model output presented as fact with no source citation, confidence indicator, or "based on retrieved documents" framing
- No retrieval grounding for factual claims (pure-generation answers in a domain that requires sources: medical, legal, financial, regulatory, news)
- Hallucinated citations / URLs / case numbers / API endpoints emitted to users with no verification step (URL existence check, citation back-resolution to the corpus)
- Fabricated package or library names in code suggestions (slopsquatting risk) - no check that suggested imports actually exist in the registry
- No "I don't know" pathway: prompt does not instruct refusal on uncertainty, sampling temperature too high for factual tasks
- Downstream systems treat LLM output as authoritative input with no validation (e.g. LLM-generated metric becomes a business KPI)
- No disclaimer or jurisdiction notice for regulated domains
- No human review gate for high-stakes outputs (clinical decision, legal filing, financial advice, irreversible business action)
- No evaluation of factual accuracy on a held-out benchmark; no monitoring of hallucination rate in production
- Anchoring / sycophancy: prior conversation context biases later answers without correction
Look in: RAG citation pipeline, output post-processing, system prompts for hallucination guards, code-suggestion handlers, dashboards or reports that consume LLM output, evaluation harness.
Fixes: require citations resolvable to the retrieved corpus, refuse-on-low-confidence prompting + lower temperature for factual tasks, package-existence check for code suggestions, domain disclaimers, human review for high-stakes outputs, hallucination-rate monitoring against a golden set, "show your sources" UI for every factual claim.
LLM10:2025 - Unbounded Consumption
Covers Denial of Service and Denial of Wallet (DoW) - attacker-driven cost explosion against your provider bill.
Check for:
- No per-user / per-session / per-IP request rate limit on LLM endpoints (Upstash, Vercel rate limit, framework middleware)
- No daily or monthly cost cap and no per-tenant token quota
- Input length unbounded: user can submit a 1MB prompt, paste an entire book, attach huge files
- Output
max_tokens not set or set to model maximum on every call (attacker forces max output)
- Multimodal: no size limit on uploaded images / audio / PDFs (image tokens scale with resolution, audio with duration)
- Streaming endpoints with no idle timeout - attacker holds connection open consuming a slot
- Recursive / agentic loops with no iteration cap, no depth cap, no wall-clock budget, no cost budget per task
- Tool calls that re-invoke the LLM (e.g. summarization tool, sub-agent tool) without contributing to the parent budget
- Cache poisoning: per-user cache key omits user identifier, attacker fills shared cache; or cache disabled entirely on identical hot prompts
- No timeout on provider HTTP calls (default SDK timeouts can be infinite or very long)
- No alerting on token-spend anomalies, no circuit breaker that disables the endpoint at a spend threshold
- Free / unauthenticated endpoints that proxy to a paid provider model
- Embedding / fine-tuning endpoints (very expensive) with no auth or rate limit
- Model selection controlled by user input (attacker requests the most expensive model)
Look in: rate-limit middleware, API route handlers calling provider SDKs, agent loop code, streaming handlers, file upload size limits, provider client config (timeouts, retries), cost-monitoring dashboards, cache key construction.
Fixes: per-user token + request rate limits, daily/monthly cost cap with circuit breaker, hard max_tokens on every call, truncate input to a known bound, file-size and duration limits on multimodal inputs, agent loop caps (iterations, depth, wall clock, cost), per-request timeouts, anomaly alerts on token spend, authenticate every paid endpoint, server-side model allowlist (not user-controlled), include user ID in cache keys.
Phase 4: Report
Skip this phase if fix-critical was selected in Phase 1 (that option requests fixes only, no report).
Otherwise, generate security/llm-top10-report.md (path relative to repo root - create the security/ directory if missing):
# OWASP LLM Top 10:2025 Security Audit
**Date:** <today>
**Architecture:** <from interview>
**Provider / SDK:** <from interview>
**Categories audited:** <list>
**Overall Risk:** Critical | High | Medium | Low
## Summary Table
| ID | Category | Severity | Findings | Status |
|----|----------|----------|----------|--------|
| LLM01 | Prompt Injection | Critical | 3 | Open |
| LLM02 | Sensitive Information Disclosure | ... | N | ... |
| LLM03 | Supply Chain | ... | N | ... |
| LLM04 | Data and Model Poisoning | ... | N | ... |
| LLM05 | Improper Output Handling | ... | N | ... |
| LLM06 | Excessive Agency | ... | N | ... |
| LLM07 | System Prompt Leakage | ... | N | ... |
| LLM08 | Vector and Embedding Weaknesses | ... | N | ... |
| LLM09 | Misinformation | ... | N | ... |
| LLM10 | Unbounded Consumption | ... | N | ... |
## Critical Findings (fix immediately)
### LLM0X-1: <short title>
- **File:** `path/to/file.ts:42`
- **Issue:** <one-sentence description>
- **Evidence:** <code snippet or pattern>
- **Impact:** <what an attacker achieves>
- **Fix:** <specific change>
- **Status:** Open | Fixed in commit `<sha>`
## High Findings (fix before next release)
...
## Medium Findings (fix within 30 days)
...
## Low Findings (track and monitor)
...
## Passing Controls
- LLM0X: <one-line description of what passed and why>
## Remediation Checklist
- [ ] LLM0X-1: ...
- [ ] LLM0X-2: ...
## Out-of-Code Items
Issues that require infrastructure, provider settings, or prompt-engineering work not in the repo:
- [ ] ...
Phase 5: Fix (if report-and-fix or fix-critical selected)
For each Critical finding (and High findings when report-and-fix is selected):
- Implement the specific fix identified in Phase 3
- Add a test or assertion that fails without the fix where feasible (regex denylist test, schema-validation unit test, rate-limit integration test)
- Mark the finding
Fixed in commit <sha> in the report
Do not change model provider, prompts, or architecture beyond what is needed to address the specific finding. If a finding requires architectural change, leave it Open and add it to Out-of-Code Items.
Completion
Output:
- The full path to the report file
- List of every file changed with a one-line description
- Findings still Open and why (out-of-code, deferred, needs infra change)
- Any new tests added
