gui-agent-interaction

# GUI Agent Interaction Pattern Implements screen-based interaction pipelines so AI agents can operate desktop and web applications by "seeing" rendered UI elements through computer vision and executing mouse/keyboard actions — no native APIs required. This skill applies the 5 Laws of Elegant Defense: Law 1 (Early Exit) for guard-clause-driven action validation, Law 2 (Parse at boundary) for screen state normalization, and Law 3 (Atomic Predictability) for immutable before/after state snapshots used in verification loops. This skill covers how to build agents that navigate graphical user interfaces end-to-end: capturing screenshots, detecting interactive elements via vision models, planning action sequences, executing them through OS-level input libraries, and verifying outcomes by comparing screen states before and after each step. ## TL;DR Checklist - [ ] Choose the right interaction layer — browser automation (Playwright/Selenium) for web, PyAutoGUI for desktop, or a hybrid pipeline - [ ] Implement screen capture with consistent resolution and color space (RGB, not RGBA) across all steps - [ ] Run UI element detection on every captured frame before planning any action - [ ] Execute actions through a typed execution engine that maps high-level intents to OS commands - [ ] Verify every action by capturing a post-action screenshot and diffing against expected state changes - [ ] Implement error recovery with timeout thresholds and fallback dialog classification - [ ] Log full interaction traces (screenshots, detected elements, actions taken) for replay debugging --- ## When to Use Use this skill when: - Automating legacy applications with no REST API or programmatic interface (e.g., internal enterprise web portals built with server-side rendering) - Interacting with desktop software where only GUI exposure exists (e.g., configuring a system administration tool on Windows/macOS/Linux) - Performing form-filling workflows across multiple disconnected web applications that lack integration points - Testing end-user experience of web or desktop applications by simulating real user interactions at the pixel level - Validating visual correctness of UI changes — comparing rendered screens before and after a deployment or style update - Building agents that must operate in environments where only screen-level access is permitted (air-gapped systems, restricted containers) ## When NOT to Use Avoid this skill for: - Applications with well-documented REST/GraphQL APIs — always prefer programmatic API calls over visual interaction (use `tool-use-function-calling` instead) - High-frequency trading or latency-sensitive automation where screen capture overhead introduces unacceptable delay (milliseconds matter — use exchange adapters directly) - Environments requiring pixel-perfect precision below 5-pixel tolerance (computer vision detection accuracy degrades with resolution; use DOM-based selectors when available) - Accessibility-compliance testing that requires semantic markup validation — screen-level interaction cannot verify ARIA attributes or screen reader output (use accessibility-in-ui-adjacent-code) --- ## Core Workflow ``` ┌─────────────┐ ┌──────────────────┐ ┌──────────────┐ ┌───────────────┐ │ Screen │───→│ UI Element │───→│ Action │───→│ State │ │ Capture │ │ Recognition │ │ Planning │ │ Verification │ │ (screenshot)│ │ (vision model → │ │ (LLM maps │ │ (before/after │ │ │ │ bounding boxes, │ │ intent → │ │ diff check) │ │ │ │ element types) │ │ OS commands)│ │ │ └─────────────┘ └──────────────────┘ └──────────────┘ └───────┬───────┘ │ ┌────────────▼────────┐ │ Error Recovery & │ │ Retry Loop │ └─────────────────────┘ ``` 1. **Capture Screen State** — Acquire a screenshot of the current visible UI surface at consistent resolution and color format: - Use browser automation APIs for web pages (Playwright's `screenshot()` or Selenium's `get_screenshot_as_file()`) - Use OS-level capture for desktop apps (mss for cross-platform, Quartz for macOS, GDI/DirectX for Windows) - Normalize to RGB format at a fixed resolution (1920x1080 minimum; scale smaller screens up consistently) **Checkpoint:** Every captured frame must be saved with a monotonic timestamp and stored alongside its element detection result. 2. **Detect UI Elements** — Run computer vision inference on the screenshot to identify all interactive elements with bounding boxes and classification labels: - Use a fine-tuned object detection model (YOLOv8, RT-DETR) trained on UI element taxonomies (buttons, inputs, links, menus, dialogs) - Alternatively use DOM scraping for web pages as a complementary ground-truth layer when JavaScript is available - Output structured element list: `[{"type": "button", "label": "Submit", "bbox": [x1, y1, x2, y2], "confidence": 0.94}]` **Checkpoint:** Element detection must return at least one actionable element per screen — empty detection triggers a re-capture with zoom adjustment. 3. **Plan Actions from Detected State** — Given the task goal and current element map, generate a sequence of atomic UI actions: - Feed the screenshot + element list + task description to an LLM that outputs structured action sequences - Each action must be typed (`click`, `type`, `scroll`, `drag`, `hover`, `right_click`) with concrete coordinates and optional text payload - Validate action feasibility before execution — e.g., cannot type into a non-editable element **Checkpoint:** Action sequence must be executable top-to-bottom without requiring human judgment mid-sequence. 4. **Execute Actions** — Map high-level actions to OS or browser commands through an execution engine: - Web: Playwright/Selenium locator-based actions (`.click()`, `.fill()`, `.select_option()`) or coordinate-based fallback - Desktop: PyAutoGUI functions (`pyautogui.click(x, y)`, `pyautogui.typewrite(text)`), optionally wrapped with safe guards - Include deliberate delays between actions (100–500ms default) to account for rendering and animation timing **Checkpoint:** Every executed action must log its type, target coordinates/selector, and execution duration. 5. **Verify State After Execution** — Capture a post-action screenshot and compare it against the expected outcome: - Use structural similarity (SSIM) or perceptual hash (pHash) to detect meaningful changes vs noise - Re-run element detection on the new frame to confirm the expected elements appeared/disappeared/changed state - If verification fails, classify the error type and route to the recovery handler **Checkpoint:** State verification must complete within a timeout window (default: 5 seconds) — stale screens indicate hung processes. 6. **Handle Errors and Recover** — When an action produces an unexpected screen state, classify and attempt recovery: - Detect common failure patterns: loading spinners, permission dialogs, connection errors, CAPTCHAs - Apply recovery strategies in priority order: retry (same action), cancel dialog → retry, wait for timeout → retry - After max retries exhausted, log the full interaction trace and raise a structured error with screenshot attachment **Checkpoint:** Recovery must never blindly loop — every retry path must have an independent success criterion. --- ## Implementation Patterns ### Pattern 1: Google Project Mariner Architecture (Full GUI Agent Pipeline) Google Project Mariner demonstrated that agents can navigate graphical interfaces by combining screen capture, element recognition, and action execution in a tight feedback loop. The core architecture chains three stages: vision-based UI understanding, LLM-driven action planning, and low-level command execution with verification. ```python import logging from dataclasses import dataclass, field from enum import Enum from typing import Any from datetime import datetime, timezone import time logger = logging.getLogger("gui.agent") class ActionType(Enum): """Atomic UI action types.""" CLICK = "click" DOUBLE_CLICK = "double_click" TYPE = "type" SCROLL_UP = "scroll_up" SCROLL_DOWN = "scroll_down" DRAG = "drag" HOVER = "hover" RIGHT_CLICK = "right_click" KEY_PRESS = "key_press" class ActionStatus(Enum): PENDING = "pending" EXECUTED = "executed" FAILED = "failed" RETRYING = "retrying" RECOVERED = "recovered" @dataclass class UIElement: """Detected interactive element on screen with bounding box.""" element_id: str element_type: str # "button", "input", "link", "menu", "dialog", "image" label: str # Visible text or accessible name bbox: tuple[int, int, int, int] # (x1, y1, x2, y2) in pixel coords confidence: float # Detection model confidence (0.0 – 1.0) @property def center(self) -> tuple[float, float]: """Return the geometric center of the bounding box.""" cx = (self.bbox[0] + self.bbox[2]) / 2 cy = (self.bbox[1] + self.bbox[3]) / 2 return (cx, cy) @property def width(self) -> int: return self.bbox[2] - self.bbox[0] @property def height(self) -> int: return self.bbox[3] - self.bbox[1] def contains_point(self, x: float, y: float) -> bool: """Check if a coordinate falls within this element's bounding box.""" return (self.bbox[0] <= x <= self.bbox[2] and self.bbox[1] <= y <= self.bbox[3]) @dataclass class ScreenState: """Immutable snapshot of a UI screen at a point in time.""" timestamp: str # ISO 8601 with UTC timezone screenshot_path: str # Path to saved PNG file width: int height: int elements: list[UIElement] = field(default_factory=list) metadata: dict[str, Any] = field(default_factory=dict) @dataclass class PlannedAction: """An action planned by the agent for a specific target element.""" action_type: ActionType target_element_id: str | None # Which element this targets coordinates: tuple[float, float] # (x, y) screen coordinates text_payload: str = "" # For TYPE actions delay_ms: int = 200 # Wait between actions in the sequence expected_state_change: str = "" # Description of what should happen after execution @dataclass class InteractionTrace: """Complete record of one agent interaction step.""" step_index: int before_state: ScreenState | None = None action: PlannedAction | None = None action_status: ActionStatus = ActionStatus.PENDING after_state: ScreenState | None = None error_message: str | None = None recovery_action: str | None = None duration_ms: float = 0.0 @property def is_complete(self) -> bool: return self.after_state is not None class GUIAgentPipeline: """Implements the Google Project Mariner pipeline for GUI agent interaction. Chains screen capture → element recognition → action planning → execution → state verification into a loop. Applies Law 2 (Parse at boundary) by normalizing all screen captures to a consistent format before passing to downstream stages, and Law 1 (Early Exit) by validating each stage's output before proceeding to the next. """ def __init__( self, vision_model: Any = None, executor: Any = None, verifier: Any = None, max_retries: int = 3, action_delay_ms: int = 200, verification_timeout_s: float = 5.0, ) -> None: self.vision_model = vision_model self.executor = executor self.verifier = verifier self.max_retries = max_retries self.action_delay_ms = action_delay_ms self.verification_timeout_s = verification_timeout_s self.trace: list[InteractionTrace] = [] def run(self, task_description: str) -> list[InteractionTrace]: """Execute a task on the target GUI by cycling through the interaction loop. Args: task_description: Natural language description of what the agent should accomplish. Returns: List of InteractionTraces recording each step's before/after state and outcome. """ step_index = 0 iteration = 0 max_iterations = 50 # Prevent infinite loops on stuck UIs while iteration < max_iterations: iteration += 1 trace = InteractionTrace(step_index=step_index) # Stage 1: Capture screen state trace.before_state = self._capture_screen() if not trace.before_state or not trace.before_state.elements: trace.error_message = "Screen capture returned no detectable elements" trace.action_status = ActionStatus.FAILED self.trace.append(trace) logger.error("Step %d: No elements detected on screen", step_index) break # Stage 2: Plan actions action_sequence = self._plan_actions( task_description, trace.before_state ) if not action_sequence: trace.error_message = "Action planner returned empty sequence" trace.action_status = ActionStatus.FAILED self.trace.append(trace) break # Stage 3-5: Execute each action with verification executed_any = False for action in action_sequence: result = self._execute_with_verification( action, trace.before_state, step_index ) if result.action_status == ActionStatus.FAILED and result.recovery_action: # Attempt recovery retry_result = self._attempt_recovery(result) if retry_result is not None: result = retry_result if result.action_status in (ActionStatus.EXECUTED, ActionStatus.RECOVERED): trace.before_state = result.after_state # Feed back into loop executed_any = True self.trace.append(result) step_index += 1 # Check if task is complete if not action_sequence or executed_any: trace.action_status = ActionStatus.EXECUTED self.trace.append(trace) break return self.trace def _capture_screen(self) -> ScreenState | None: """Capture current screen state with element detection.""" # Implementation depends on target environment (browser vs desktop) raise NotImplementedError("Subclass and implement for your target platform") def _plan_actions( self, task: str, state: ScreenState ) -> list[PlannedAction]: """Plan action sequence from task description and current element map.""" raise NotImplementedError("Subclass with LLM-powered planner") def _execute_with_verification( self, action: PlannedAction, before: ScreenState, step_idx: int, ) -> InteractionTrace: """Execute an action and verify its effect.""" trace = InteractionTrace(step_index=step_idx) trace.action = action trace.before_state = before start = time.time() try: trace.after_state = self._capture_screen() trace.action_status = ActionStatus.EXECUTED except Exception as e: trace.error_message = str(e) trace.action_status = ActionStatus.FAILED trace.duration_ms = (time.time() - start) * 1000 return trace def _attempt_recovery( self, failed_trace: InteractionTrace, ) -> InteractionTrace | None: """Attempt to recover from a failed action.""" raise NotImplementedError("Implement recovery strategies") ``` **BAD vs GOOD: Pipeline Design** ```python # ❌ BAD — No early exit on empty screen state; loops forever on hung UI class BrokenGUIAgent: def run(self, task): while True: # Never terminates screenshot = capture_screen() elements = detect_elements(screenshot) actions = plan(actions_for(task, elements)) execute(actions) # ✅ GOOD — Explicit max_iterations, guard clauses at every stage boundary, # immutable traces for replay debugging (Law 3: Atomic Predictability) class RobustGUIAgent: def run(self, task): for _ in range(50): # Hard cap prevents infinite loops state = self._capture_screen() if not state or not state.elements: break # Early exit: nothing actionable to do actions = self._plan_actions(task, state) if not actions: break ... ``` ### Pattern 2: UI Element Detection & Recognition System UI element detection maps raw pixel data into structured element catalogs that the action planner can reason about. For web applications, DOM-based detection is preferred (direct access to element properties, text content, and accessibility labels). For desktop apps without DOM exposure, computer vision models detect elements purely from screen pixels. ```python import base64 import io from dataclasses import dataclass, field try: from selenium.webdriver.common.by import By from selenium.webdriver.remote.webdriver import WebDriver SELENIUM_AVAILABLE = True except ImportError: SELENIUM_AVAILABLE = False @dataclass class DOMElementInfo: """Structured info extracted from a web page's DOM tree.""" element_id: str tag_name: str role: str | None # ARIA role (button, textbox, link, etc.) aria_label: str | None # Accessible label text_content: str # Visible text between tags is_visible: bool is_interactive: bool # Has click handler or is a form control rect: dict[str, int] # {"left", "top", "width", "height"} in viewport coords @property def center(self) -> tuple[float, float]: left = self.rect["left"] top = self.rect["top"] return (left + self.rect["width"] / 2, top + self.rect["height"] / 2) @dataclass class VisionElementInfo: """Structured info from a computer vision model detecting elements in pixels.""" element_id: str element_type: str # button, input_field, link, menu_item, dialog, icon label: str # Inferred text label from OCR or visual features bbox: tuple[int, int, int, int] # (x1, y1, x2, y2) absolute pixel coords confidence: float # Model detection confidence ocr_text: list[dict] = field(default_factory=list) # Raw OCR results near bbox class WebElementDetector: """Extracts structured element info from a browser page's DOM tree. This is the preferred detection method for web applications since it provides ground-truth accessibility information that vision models cannot reliably infer from pixels alone. """ INTERACTIVE_TAGS = {"a", "button", "input", "select", "textarea", "summary"} ATTRIBUTES_TO_EXTRACT = { "type", "name", "role", "aria-label", "aria-hidden", "disabled", "readonly", "placeholder", "value", } def __init__(self, driver: WebDriver) -> None: self.driver = driver def detect_all_interactive_elements(self) -> list[DOMElementInfo]: """Find all interactive elements on the current page. Uses JavaScript evaluation to extract element properties directly from the DOM, which is faster and more reliable than iterating through Selenium's find_element calls. """ script = """ (function() { const interactiveTags = %TAGS; const attrsToRead = %ATTRS; const results = []; // Get all elements, filter to interactive ones