design-llm-plugin-security

name	design-llm-plugin-security
description	Use when building tools, plugins, or function-calling integrations for LLM systems — defining what actions the model can invoke, how those invocations are validated, and how to prevent misuse of plugin capabilities.
source	OWASP Top 10 for LLM Applications 2025 LLM07 (owasp.org/www-project-top-10-for-large-language-model-applications/); NIST AI RMF 1.0 Govern 1.3; Anthropic tool use documentation; OpenAI function calling documentation
tags	["security","owasp","llm","plugins","tool-use","function-calling","ai-security","emerging"]
emerging	true

Design LLM Plugin Security

Build LLM tool and plugin interfaces with explicit input validation, minimal permissions, human confirmation for consequential actions, and sandboxed execution — preventing prompt injection from triggering unauthorized tool invocations.

Why This Is Best Practice

Adopted by: OWASP Top 10 for LLM Applications 2025 LLM07 (Insecure Plugin Design). OpenAI, Anthropic, and Google's function calling documentation all include security guidance for tool definitions. The Model Context Protocol (MCP) specification from Anthropic includes authorization requirements for tool invocations. NIST AI RMF 1.0 Govern 1.3 requires human oversight mechanisms for automated AI systems. Status: Emerging — function calling / tool use became mainstream in 2023; security standards are still being developed. Impact: LLM plugins that invoke APIs, execute code, or modify data are the primary escalation path from prompt injection (information disclosure) to action execution (data deletion, unauthorized transactions, system compromise). Demonstrated attacks: manipulated AI email assistants sent emails to contacts, AI code assistants triggered malicious package installs via injected code comments, AI financial assistants initiated unauthorized transfers via forged instructions in documents. Why best: Trusting the LLM to only call tools when appropriate is the common approach — it fails whenever the LLM is manipulated via prompt injection. Explicit authorization checks, input validation, and confirmation for high-impact actions provide defense-in-depth that holds even when the LLM is injected.

Sources: OWASP LLM Top 10 2025 LLM07; NIST AI RMF 1.0; MCP specification; OpenAI function calling security guide

Steps

1. Define strict, narrow tool schemas — minimize what each tool can accept:

   tools = [
       {
           "name": "search_documents",
           "description": "Search the user's own documents. Only returns documents owned by the authenticated user.",
           "input_schema": {
               "type": "object",
               "properties": {
                   "query": {
                       "type": "string",
                       "maxLength": 500,  # hard limit
                       "description": "Search query"
                   }
               },
               "required": ["query"],
               "additionalProperties": False  # reject unexpected params
           }
       }
   ]
   

2. Validate all tool inputs before execution — never pass LLM-generated parameters directly to tools:

   from pydantic import BaseModel, validator, constr
   
   class SearchDocumentsInput(BaseModel):
       query: constr(min_length=1, max_length=500)
   
       @validator('query')
       def no_injection_patterns(cls, v):
           # Reject obvious injection patterns
           suspicious = ['ignore previous', 'system prompt', 'act as root']
           for pattern  suspicious:
                pattern.lower()  v.lower():
                    ValueError()
            v
   
    ():
       
        tool_name == :
           params = SearchDocumentsInput(**raw_params)
            search_documents(params.query, owner_id=user_id)  
        ValueError()
   

Rules

• Never pass raw LLM tool parameters to backend functions — always parse through a typed schema first.
• Tool descriptions in the schema influence what the LLM calls them for — be precise: "Search the authenticated user's documents" not "Search documents".
• Sandboxed code execution tools (Python interpreter, shell) are the highest-risk tool type — require the most restrictive containment (no network, no filesystem outside /tmp, resource limits).
• Plugin chains (tool A calls tool B) multiply the attack surface — audit each tool independently and require authorization at each hop.

Common Mistakes

• Exposing all tools for all tasks — the LLM sees what it can call; fewer tools = smaller injection target.
• Not validating that tool output is within the user's access scope — a search tool may return documents from other users if ownership filtering is in the DB query, not the tool schema.
• Treating tool call parameters as safe because they came from the LLM — they came from user-influenced context; treat them as untrusted.
• No audit log for tool invocations — agentic systems need forensic trails; "the agent did it" is not sufficient incident response.