design-llm-action-boundaries

name	design-llm-action-boundaries
description	Use when building LLM agents that can take actions in the world — sending emails, modifying files, making API calls, or executing code — to prevent the model from taking actions beyond its intended scope.
source	OWASP Top 10 for LLM Applications 2025 LLM08 (owasp.org/www-project-top-10-for-large-language-model-applications/); NIST AI RMF 1.0 Govern 1.3; EU AI Act Article 14; CWE-284
tags	["security","owasp","llm","agents","excessive-agency","action-boundaries","ai-safety","emerging"]
emerging	true

Design LLM Action Boundaries

Constrain LLM agents to the minimum necessary capabilities, enforce human-in-the-loop for irreversible actions, and implement hard permission limits that cannot be overridden by prompt — preventing agents from acting outside their intended scope.

Why This Is Best Practice

Adopted by: OWASP Top 10 for LLM Applications 2025 LLM08 (Excessive Agency). NIST AI RMF 1.0 (2023) Govern 1.3 requires human oversight for autonomous AI systems. EU AI Act Article 14 (2024) mandates human oversight mechanisms for high-risk AI. Anthropic's Model Specification, OpenAI's GPT policies, and Google's Responsible AI Practices all include capability limitation guidance for agentic systems. Status: Emerging — agentic AI (AI taking actions autonomously) became mainstream in 2023; safety standards are still maturing. This is one of the most rapidly evolving areas in AI safety. Impact: AutoGPT and BabyAGI (2023) were demonstrated deleting files, sending emails, and making API calls based on prompt injections from web pages. AI coding assistants have been demonstrated installing malicious packages via injected code comments. An AI customer service agent with CRM write access and no approval requirement was manipulated into issuing fraudulent refunds. The more actions an agent can take, the higher the potential blast radius of a successful injection. Why best: Reactive monitoring (detecting unauthorized actions after they occur) is the alternative — it cannot undo irreversible actions like sent emails, deleted data, or executed financial transactions. Proactive capability limitation prevents the actions from occurring in the first place.

Sources: OWASP LLM Top 10 2025 LLM08; NIST AI RMF 1.0 Govern 1.3; EU AI Act Article 14; AutoGPT security research (2023)

Steps

1. Apply strict least-privilege to agent capabilities:

   from enum import Enum
   
   class AgentCapability(Enum):
       READ_DOCS = "read_docs"
       WRITE_DOCS = "write_docs"
       SEND_EMAIL = "send_email"
       EXECUTE_CODE = "execute_code"
       CALL_EXTERNAL_API = "call_external_api"
   
   # Task-specific capability grants — grant only what the task needs
   TASK_CAPABILITIES = {
       'document_summary': {AgentCapability.READ_DOCS},
       'draft_email': {AgentCapability.READ_DOCS},  # NOT send_email
       'send_approved_email': {AgentCapability.SEND_EMAIL},
       'code_review': {AgentCapability.READ_DOCS, AgentCapability.EXECUTE_CODE},
   }
   
   class BoundedAgent:
       def __init__(self, task_type: str, user_id: str):
           self.capabilities = TASK_CAPABILITIES.get(task_type, set())
           self.user_id = user_id
   
       def can_do(self, capability: AgentCapability) -> bool:
           return capability in self.capabilities
   

2. Separate planning from execution — require approval for action plans:

   def two_phase_execution(agent, task: str):
       # Phase 1: Generate action plan (read-only)
       plan = agent.plan(task)  
   
       
       approved_actions = present_plan_for_approval(plan)
   
        approved_actions  :
            {: , : }
   
       
       results = []
        action  approved_actions:
           result = agent.execute_single_action(action)
           results.append(result)
   
        {: , : results}
   

Rules

• The LLM should plan; deterministic code should execute — never let the LLM execute actions directly without a typed execution layer.
• Irreversible action gates must be implemented outside the LLM's influence — hard-coded in application logic, not enforced via system prompt.
• An agent's capabilities should be tied to the specific task, not the user's maximum permissions — a document-summarization task doesn't need email sending capability even if the user has that permission.
• The more autonomous the agent, the more restrictive the capability set should be — fully autonomous agents need the tightest constraints.

Common Mistakes

• Giving agents the same permissions as the user — users have permissions for interactive tasks; agents acting autonomously should have a subset.
• Using the system prompt as the only capability gate — "only read files, never delete" in a system prompt is bypassed by injection; hard code the gate.
• No human-in-the-loop for multi-step agentic tasks — each step can compound errors or malicious actions; checkpoints limit blast radius.
• Logging only errors, not successful actions — forensics requires knowing what the agent did, not just when it failed.