mobile-agent-v3-gui-automation

name	mobile-agent-v3-gui-automation
title	Mobile-Agent-v3: Foundational GUI Automation with Self-Evolving Trajectories
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.15144
keywords	["gui-automation","mobile-agents","trajectory-generation","reinforcement-learning","desktop-automation"]
description	Build GUI automation agents using self-evolving trajectory generation, trajectory-aware policy optimization, and integrated action semantics for cross-platform interaction.

Mobile-Agent-v3: Foundational GUI Automation with Self-Evolving Trajectories

Core Concept

Mobile-Agent-v3 enables robust GUI automation across desktop and mobile platforms by combining self-evolving trajectory generation, integrated agent capabilities (UI grounding, planning, action semantics), and trajectory-aware reinforcement learning. The system generates high-quality interaction data automatically through iterative refinement, eliminating manual annotation. GUI-Owl, the foundational model, grounds visual understanding with semantic action capabilities, achieving state-of-the-art performance on diverse benchmarks.

Architecture Overview

• Self-Evolving Trajectory Production: Cloud infrastructure for autonomous interaction data generation
• GUI-Owl Foundation Model: Integrated visual grounding with action semantics
• Trajectory-Aware Relative Policy Optimization (TRPO): RL training with trajectory awareness
• Multi-Platform Support: Android, Windows, macOS, Linux execution
• Modular Agent Capabilities: UI grounding, planning, action semantics, reasoning

Implementation Steps

1. Implement GUI State Representation

Create unified abstraction for visual and semantic information:

from dataclasses import dataclass
from typing import List, Tuple, Dict, Any
import cv2
import numpy as np

@dataclass
class GUIElement:
    """Represents interactive element in GUI."""
    element_id: str
    bbox: Tuple[int, int, int, int]  # x1, y1, x2, y2
    element_type: str  # button, text_input, dropdown, etc.
    text_content: str
    clickable: bool
    visible: bool
    parent_id: str = None
    semantic_role: str = None  # based on context

@dataclass
class GUIState:
    """Complete GUI state snapshot."""
    screenshot: np.ndarray  # RGB image
    timestamp: float
    platform: str  # android, windows, macos, linux
    elements: List[GUIElement]
    dom_tree: Dict[str, Any]  # Hierarchical element structure
    text_content: str  # OCR extracted text

    def get_clickable_elements() -> [GUIElement]:
         [e  e  .elements  e.clickable  e.visible]

     () -> GUIElement:
         ((e  e  .elements  e.element_id == element_id), )

2. Implement Action Semantics

Define action space and execution:

from enum import Enum

class ActionType(Enum):
    CLICK = "click"
    SCROLL = "scroll"
    LONG_PRESS = "long_press"
    SWIPE = "swipe"
    TYPE_TEXT = "type_text"
    KEY_PRESS = "key_press"
    DRAG = "drag"

@dataclass
class Action:
    """Semantic action representation."""
    action_type: ActionType
    target_element_id: str = None
    parameters: Dict[str, Any] = None  # x, y, text, direction, duration, etc.

    def to_execution_command(self, gui_state: GUIState) -> Dict[str, Any]:
        """Convert semantic action to platform-specific command."""
        element = gui_state.get_element_by_id(self.target_element_id) if self.target_element_id else None

        if self.action_type == ActionType.CLICK:
            x, y = self._get_click_coordinates(element)
            return {"type": "click", "x": x, "y": y}

        elif self.action_type == ActionType.SCROLL:
            direction = self.parameters.get(, )
            amount = .parameters.get(, )
             {: , : direction, : amount}

         .action_type == ActionType.TYPE_TEXT:
            text = .parameters.get(, )
             {: , : text}

         {}

     () -> [, ]:
         element:
            x1, y1, x2, y2 = element.bbox
             ((x1 + x2) // , (y1 + y2) // )
        :
             (.parameters.get(, ), .parameters.get(, ))

3. Implement UI Grounding Module

Ground visual elements with semantic understanding:

import torch
from typing import Callable

class UIGrounder:
    """Grounds visual elements to semantic understanding."""

    def __init__(self, vision_model: Callable, language_model: Callable):
        self.vision_model = vision_model
        self.language_model = language_model

    def ground_elements(
        self,
        gui_state: GUIState,
        task_description: str
    ) -> List[Tuple[GUIElement, float, str]]:
        """
        Ground elements: (element, relevance_score, semantic_role)
        """
        results = []

        for element in gui_state.get_clickable_elements():
            # Extract visual features around element
            x1, y1, x2, y2 = element.bbox
            element_patch = gui_state.screenshot[y1:y2, x1:x2]

            # Get visual embedding
            visual_embedding = self.vision_model.encode(element_patch)

            # Create semantic context
            context = f"Task: {task_description}\nElement text: {element.text_content}\nType: {element.element_type}"

            # Compute relevance
            semantic_embedding = self.language_model.encode(context)
            relevance = torch.cosine_similarity(
                torch.tensor(visual_embedding).unsqueeze(0),
                torch.tensor(semantic_embedding).unsqueeze(0)
            ).item()

            # Determine semantic role
            role = ._infer_semantic_role(element, task_description, context)

            results.append((element, relevance, role))

        
        results.sort(key= x: x[], reverse=)
         results

     () -> :
        
        
        role_prompt = 
        role = .language_model.generate(role_prompt, max_tokens=)
         role

4. Implement Trajectory Collection and Evolution

Generate and iteratively improve interaction trajectories:

@dataclass
class Trajectory:
    """Complete interaction sequence."""
    task_description: str
    initial_state: GUIState
    actions: List[Action]
    states: List[GUIState]
    rewards: List[float]
    success: bool
    task_completion_rate: float

class TrajectoryCollector:
    def __init__(self, executor: "GUIExecutor", evaluator: "TaskEvaluator"):
        self.executor = executor
        self.evaluator = evaluator

    def collect_trajectory(
        self,
        task_description: str,
        max_steps: int = 50,
        initial_state: GUIState = None
    ) -> Trajectory:
        """Execute task and collect trajectory."""

        if initial_state is None:
            initial_state = self.executor.get_current_state()

        current_state = initial_state
        states = [initial_state]
        actions = []
        rewards = []

        for step in range(max_steps):
            # Generate action (initially: rule-based or random)
            action = self._generate_action(task_description, current_state, step)
            actions.append(action)

            # Execute action
            current_state = self.executor.execute_action(action)
            states.append(current_state)

            # Evaluate progress
            reward = .evaluator.compute_reward(
                task_description,
                current_state,
                previous_state=states[-]  (states) >   
            )
            rewards.append(reward)

            
             .evaluator.is_task_complete(task_description, current_state):
                

        
        success = .evaluator.is_task_complete(task_description, current_state)
        completion_rate = .evaluator.compute_completion_rate(
            task_description, current_state
        )

         Trajectory(
            task_description=task_description,
            initial_state=initial_state,
            actions=actions,
            states=states,
            rewards=rewards,
            success=success,
            task_completion_rate=completion_rate
        )

     () -> Action:
        
        
        
        

     () -> [Trajectory]:
        
        evolved = [base_trajectory]

         _  (num_variations):
            
            modified_actions = ._perturb_actions(base_trajectory.actions)

            
            
            new_traj = .collect_trajectory(
                base_trajectory.task_description
            )

             new_traj.task_completion_rate > base_trajectory.task_completion_rate:
                evolved.append(new_traj)

         evolved

     () -> [Action]:
        
        
        

5. Implement Trajectory-Aware Policy Optimization (TRPO)

Fine-tune agents using collected trajectories:

class TrajectoryAwarePolicy:
    """Policy model aware of trajectory context."""

    def __init__(self, model: torch.nn.Module, learning_rate: float = 1e-5):
        self.model = model
        self.optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

    def compute_trajectory_loss(
        self,
        trajectory: Trajectory,
        gamma: float = 0.99
    ) -> torch.Tensor:
        """
        Compute loss considering entire trajectory context.
        """
        # Compute discounted cumulative rewards
        returns = []
        cumulative = 0
        for reward in reversed(trajectory.rewards):
            cumulative = reward + gamma * cumulative
            returns.insert(0, cumulative)

        returns = torch.tensor(returns, dtype=torch.float32)

        # Normalize returns
        returns = (returns - returns.mean()) / (returns.std() + 1e-8)

        total_loss = 0.0

        for step, (state, action, ret) in enumerate(zip(
            trajectory.states[:-1],
            trajectory.actions,
            returns
        )):
            # Get model prediction for this state
            with torch.no_grad():
                state_repr = self._encode_state(state)

            logits = self.model(state_repr)
            log_prob = torch.log_softmax(logits, dim=-1)
            action_idx = self._encode_action(action)

            
            policy_loss = -log_prob[action_idx] * ret

            
            
             step > :
                prev_state_repr = ._encode_state(trajectory.states[step - ])
                trajectory_consistency = torch.nn.functional.cosine_similarity(
                    state_repr.unsqueeze(),
                    prev_state_repr.unsqueeze()
                )
                regularization = -trajectory_consistency * 

                policy_loss = policy_loss + regularization

            total_loss += policy_loss

         total_loss / (trajectory.actions)

     ():
        
        total_loss = 

         trajectory  trajectories:
            loss = .compute_trajectory_loss(trajectory)
            .optimizer.zero_grad()
            loss.backward()
            torch.nn.utils.clip_grad_norm_(.model.parameters(), )
            .optimizer.step()

            total_loss += loss.item()

         total_loss / (trajectories)

     () -> torch.Tensor:
        
        
        visual_feat = ._extract_visual_features(state.screenshot)
        semantic_feat = ._extract_semantic_features(state)
         torch.cat([visual_feat, semantic_feat], dim=-)

     () -> :
        
        

     () -> torch.Tensor:
        

     () -> torch.Tensor:
        

Practical Guidance

When to Use Mobile-Agent-v3

• GUI automation across desktop and mobile platforms
• Task learning from limited demonstrations
• Interactive systems requiring visual understanding
• Scenarios with procedural task variations
• Production deployment of GUI agents

When NOT to Use

• Tasks without clear visual interface
• Real-time systems with strict latency (<100ms)
• Scenarios requiring semantic understanding beyond UI
• Proprietary or closed systems without API access

Key Hyperparameters

• max_steps_per_trajectory: 30-100 based on task complexity
• trajectory_variations: 3-10 for evolution
• learning_rate: 1e-5 to 1e-4
• gamma (discount factor): 0.99 standard
• trajectory_consistency_weight: 0.01-0.1

Performance Expectations

• AndroidWorld: 73.3% success rate
• OSWorld: 37.7% success rate
• Cross-platform Generalization: Strong
• Data Efficiency: High-quality trajectories reduce annotation

Reference

Researchers. (2024). Mobile-Agent-v3: Foundational Agents for GUI Automation. arXiv preprint arXiv:2508.15144.