Protects computer use agents from prompt injection by using single-shot execution planning that generates complete control flow graphs before UI observation, preventing instruction hijacking while maintaining 57% performance on frontier models.
Protects computer use agents from prompt injection by using single-shot execution planning that generates complete control flow graphs before UI observation, preventing instruction hijacking while maintaining 57% performance on frontier models.
Overview
Implement a security architecture for agents that control computers through UI interaction. Rather than observing the screen and deciding actions reactively, use single-shot planning where a trusted planner generates the complete execution graph with conditional branches before exposure to potentially malicious UI content.
When to Use
For agents that control computer interfaces autonomously
In adversarial environments where UI content may contain injection attacks
When protecting against credential theft or financial fraud through UI manipulation
For high-security applications where execution path integrity is critical
When NOT to Use
For interactive agents that must adapt based on real-time UI changes
When the full task state cannot be predetermined
For exploratory agents discovering tasks dynamically
In environments where UI layouts are highly unpredictable
Key Technical Components
Single-Shot Execution Planning
Generate complete execution graphs before observing any potentially malicious UI.
# Single-shot plan generationclassExecutionPlan:
def__init__(self, task_description):
self.task = task_description
self.control_flow_graph = Noneself.decision_points = []
defgenerate_plan(self, trusted_context=None):
"""Create complete execution graph with conditional branches"""# Generate without UI observation
plan = self.llm_generate_plan(self.task, context=trusted_context)
# Parse into control flow graphself.control_flow_graph = parse_control_flow(plan)
self.decision_points = extract_decision_nodes(self.control_flow_graph)
.control_flow_graph
():
current_node = .control_flow_graph.current_node
current_node .decision_points:
decision = .evaluate_decision(current_state)
.control_flow_graph.branch(decision)
:
.control_flow_graph.next_action()
Ensure agent cannot deviate from pre-planned execution paths.
# CFI enforcement mechanismclassControlFlowIntegrity:
def__init__(self, execution_plan):
self.plan = execution_plan
self.current_path = execution_plan.control_flow_graph
self.executed_actions = []
defvalidate_and_execute(self, proposed_action):
"""Verify action matches plan before execution"""
valid_actions = self.current_path.valid_next_actions()
if proposed_action notin valid_actions:
raise ExecutionViolation(
f"Action {proposed_action} not in pre-planned path"
)
# Execute in controlled environment
result = execute_with_containment(proposed_action)
self.executed_actions.append((proposed_action, result))
self.current_path = self.current_path.next(proposed_action)
return result
defget_execution_path(self):
"""Return verified execution trace"""returnself.executed_actions
Conditional Branch Management
Handle dynamic branching within the pre-planned graph.
# Conditional branchingclassConditionalBranch:
def__init__(self, condition, true_branch, false_branch):
self.condition = condition
self.true_branch = true_branch
self.false_branch = false_branch
defevaluate(self, state):
"""Evaluate condition using trusted state, not UI observation"""# Use internal state representation, not screen contentreturnself.condition.evaluate(state)
defexecute(self, state):
"""Execute correct branch based on condition"""ifself.evaluate(state):
returnself.true_branch
else:
returnself.false_branch
# Example: Safe conditional execution
plan = ConditionalBranch(
condition=lambda state: state["balance"] > 1000,
true_branch=["withdraw_500", "confirm_transaction"],
false_branch=["show_insufficient_funds_error"]
)
Branch Steering Attack Detection
Identify UI-based attacks attempting to force unintended branches.
# Attack detectionclassBranchSteeringDetector:
def__init__(self):
self.expected_outcomes = {}
self.suspicious_actions = []
defcheck_branch_steering(self, proposed_branch, ui_observation):
"""Detect if UI is attempting to steer execution"""
ui_elements = parse_ui_elements(ui_observation)
# Check for suspicious UI patterns
suspicious_patterns = [
"overlay_elements",
"hidden_buttons",
"obfuscated_text",
"unusual_layout"
]
for pattern in suspicious_patterns:
if detect_pattern(ui_elements, pattern):
self.suspicious_actions.append({
"timestamp": time.now(),
"pattern": pattern,
"observation": ui_observation
})
returnTruereturnFalsedefget_threat_assessment(self):
"""Assess likelihood of active steering attack"""iflen(self.suspicious_actions) > THRESHOLD:
return"high_risk"return"normal"
Trusted State Management
Maintain internal state representation separate from potentially malicious UI.
# Trusted internal stateclassTrustedState:
def__init__(self):
self.internal_model = {}
self.ui_observation = Nonedefupdate_from_reliable_source(self, source, data):
"""Update state from trusted sources only"""if is_trusted_source(source):
self.internal_model.update(data)
else:
# Log but don't trustself.log_untrusted_update(source, data)
defevaluate_condition(self, condition):
"""Use internal model, not UI observation"""return condition(self.internal_model)
defobserve_ui(self, screenshot):
"""Store UI observation for logging, not for decision-making"""self.ui_observation = screenshot
# Do NOT update internal state based on UI observation
