| name | gui-actor-grounding |
| title | GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2506.03143 |
| keywords | ["visual grounding","GUI agents","action detection","vision transformers","screen understanding"] |
| description | Enable GUI agents to ground actions without generating pixel coordinates by using attention-based patch-level alignment and a verifier for selecting optimal action regions from candidates. |
GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents
Core Concept
GUI-Actor addresses a fundamental mismatch in VLM-powered GUI agents: generating precise pixel coordinates for screen actions. Traditional approaches output (x, y) coordinates, but Vision Transformers work with patch-level features, creating a gap between dense screen pixels and sparse patch embeddings.
GUI-Actor eliminates coordinate generation entirely. Instead, it uses an attention mechanism to align a dedicated <ACTOR> token with relevant visual patches, enabling patch-based action grounding. A separate verifier selects the optimal action region from candidates. This approach achieves superior performance (44.6 on ScreenSpot-Pro vs. 38.1 for larger competitors) while maintaining frozen VLM backbones.
Architecture Overview
- Attention-Based Action Head: Align
<ACTOR> token with screen patches using cross-attention
- Patch-Level Grounding: Work directly with ViT patch granularity rather than dense pixels
- Grounding Verifier: Evaluate and rank action region candidates for selection
- Frozen VLM Backbone: Preserve pretrained vision-language model weights
- Multi-Action Support: Generate multiple action proposals in parallel
- Minimal Fine-Tuning: Only ~100M parameters trainable; rest frozen
Implementation
The following steps outline how to implement coordinate-free visual grounding for GUI agents:
- Extract patch embeddings - Get Vision Transformer patch features from the screen image
- Create action candidates - Generate multiple action proposals using the attention mechanism
- Score with verifier - Evaluate candidate quality and select optimal actions
- Execute action - Perform the selected action without explicit coordinates
- Process feedback - Update understanding based on action outcomes
- Iterate - Continue grounding new actions as agent progresses through task
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import List, Tuple, Optional
class PatchExtractor:
"""Extract patch embeddings from Vision Transformer."""
def __init__(self, patch_size: int = 16, image_size: int = 512):
self.patch_size = patch_size
self.image_size = image_size
self.num_patches = (image_size // patch_size) ** 2
def extract(self, image: torch.Tensor, vit_model) -> torch.Tensor:
"""Extract patch embeddings from image."""
with torch.no_grad():
patches = vit_model.get_patch_embeddings(image)
return patches
class ActionHead(nn.Module):
"""Attention-based action head for GUI grounding."""
def __init__(self, embedding_dim: int = 768, num_patches: int = 1024, num_actions: int = 5):
super().__init__()
self.embedding_dim = embedding_dim
self.num_patches = num_patches
.num_actions = num_actions
.actor_token = nn.Parameter(torch.randn(, , embedding_dim))
.actor_attention = nn.MultiheadAttention(embedding_dim, num_heads=, batch_first=)
.action_mlp = nn.Sequential(
nn.Linear(embedding_dim, ),
nn.ReLU(),
nn.Linear(, num_patches)
)
() -> [torch.Tensor, torch.Tensor]:
batch_size = patch_embeddings.shape[]
actor = .actor_token.expand(batch_size, -, -)
actor_attended, _ = .actor_attention(actor, patch_embeddings, patch_embeddings)
action_logits = .action_mlp(actor_attended.squeeze())
action_logits, actor_attended.squeeze()
(nn.Module):
():
().__init__()
.scoring_network = nn.Sequential(
nn.Linear(embedding_dim * , ),
nn.ReLU(),
nn.Dropout(),
nn.Linear(, ),
nn.Sigmoid()
)
() -> [torch.Tensor, []]:
batch_size, num_patches, _ = patch_embeddings.shape
top_logits, top_indices = torch.topk(action_logits, k=top_k, dim=)
verified_scores = []
b (batch_size):
scores = []
idx top_indices[b]:
patch_emb = patch_embeddings[b, idx, :]
combined = torch.cat([actor_embedding[b], patch_emb])
score = .scoring_network(combined)
scores.append(score.item())
verified_scores.append(scores)
verified_scores = torch.tensor(verified_scores)
verified_scores, top_indices
() -> torch.Tensor:
best_indices = torch.argmax(verified_scores, dim=)
selected_patches = torch.gather(patch_indices, , best_indices.unsqueeze())
selected_patches
(nn.Module):
():
().__init__()
.vit = vit_model
.action_head = ActionHead(embedding_dim, num_patches)
.verifier = GroundingVerifier(embedding_dim)
.patch_extractor = PatchExtractor()
() -> [, ]:
torch.no_grad():
patch_embeddings = .vit.get_patch_embeddings(image)
action_logits, actor_embedding = .action_head(patch_embeddings)
verified_scores, top_indices = .verifier.verify_candidates(
actor_embedding, patch_embeddings, action_logits, top_k=
)
selected_patch = .verifier.select_action(verified_scores, top_indices)
selected_patch.squeeze(), verified_scores.().item()
() -> [torch.Tensor, torch.Tensor]:
batch_size = images.shape[]
torch.no_grad():
patch_embeddings = .vit.get_patch_embeddings(images)
action_logits, actor_embedding = .action_head(patch_embeddings)
verified_scores, top_indices = .verifier.verify_candidates(
actor_embedding, patch_embeddings, action_logits, top_k=
)
selected_patches = .verifier.select_action(verified_scores, top_indices)
selected_patches, verified_scores
() -> [, ]:
patches_per_side = image_size // patch_size
row = (patch_id // patches_per_side) * patch_size + patch_size //
col = (patch_id % patches_per_side) * patch_size + patch_size //
col, row
Practical Guidance
Fine-tuning strategy:
- Frozen backbone: Keep VLM frozen during training; only update action head (~100M params)
- Learning rate: 1e-3 to 1e-4 for minimal backbone disturbance
- Batch size: 32-64 sufficient given limited trainable parameters
- Epochs: 3-10 epochs typically converge; monitor validation performance
Verifier configuration:
- Top-k candidates: 3-5 candidates; too few limits options, too many adds noise
- Scoring network depth: 2-3 layers; balance expressiveness with overfitting risk
- Confidence threshold: 0.7+ for filtering low-confidence actions
When to use:
- GUI automation and interaction tasks
- Mobile app testing and automation
- Web scraping requiring visual understanding
- Interactive systems where coordinate precision is difficult
- Transfer learning to new interfaces without coordinate retraining
When NOT to use:
- Pixel-level precision required (e.g., precise drawing tools)
- Dynamic interfaces with rapidly changing layouts
- Tasks where coordinate output is explicitly needed downstream
- Real-time systems where attention computation overhead is prohibitive
Common pitfalls:
- Frozen VLM limitation: Backbone features may not be optimal for action grounding; consider light fine-tuning
- Patch resolution tradeoff: Larger patches lose fine-grained localization; smaller patches increase computation
- Verifier collapse: Scoring network may converge to trivial solutions; use diverse training data
- Generalization: Models may overfit to specific UI layouts; validate on diverse interfaces
- Confidence calibration: Verified scores may not reflect true reliability; apply temperature scaling
Reference
GUI-Actor achieves 44.6 on ScreenSpot-Pro with Qwen2.5-VL, surpassing larger models (UI-TARS-72B at 38.1). The approach improves generalization to unseen resolutions and demonstrates effective capability transfer from frozen VLMs.
Original paper: "GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents" (arxiv.org/abs/2506.03143)
