adu2021 GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents
Core Concept
GUI-Actor addresses a fundamental mismatch in VLM-powered GUI agents: generating precise pixel coordinates for screen actions. Traditional approaches output (x, y) coordinates, but Vision Transformers work with patch-level features, creating a gap between dense screen pixels and sparse patch embeddings.
GUI-Actor eliminates coordinate generation entirely. Instead, it uses an attention mechanism to align a dedicated <ACTOR> token with relevant visual patches, enabling patch-based action grounding. A separate verifier selects the optimal action region from candidates. This approach achieves superior performance (44.6 on ScreenSpot-Pro vs. 38.1 for larger competitors) while maintaining frozen VLM backbones.
Architecture Overview
- Attention-Based Action Head: Align
<ACTOR>token with screen patches using cross-attention - Patch-Level Grounding: Work directly with ViT patch granularity rather than dense pixels
- Grounding Verifier: Evaluate and rank action region candidates for selection
- Frozen VLM Backbone: Preserve pretrained vision-language model weights
- Multi-Action Support: Generate multiple action proposals in parallel
- Minimal Fine-Tuning: Only ~100M parameters trainable; rest frozen
Implementation
The following steps outline how to implement coordinate-free visual grounding for GUI agents:
- Extract patch embeddings - Get Vision Transformer patch features from the screen image
- Create action candidates - Generate multiple action proposals using the attention mechanism
- Score with verifier - Evaluate candidate quality and select optimal actions
- Execute action - Perform the selected action without explicit coordinates
- Process feedback - Update understanding based on action outcomes
- Iterate - Continue grounding new actions as agent progresses through task
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import List, Tuple, Optional
class PatchExtractor:
"""Extract patch embeddings from Vision Transformer."""
def __init__(self, patch_size: int = 16, image_size: int = 512):
self.patch_size = patch_size
self.image_size = image_size
self.num_patches = (image_size // patch_size) ** 2
def extract(self, image: torch.Tensor, vit_model) -> torch.Tensor:
"""Extract patch embeddings from image."""
with torch.no_grad():
# Get patch embeddings from ViT
patches = vit_model.get_patch_embeddings(image)
return patches
class ActionHead(nn.Module):
"""Attention-based action head for GUI grounding."""
def __init__(self, embedding_dim: int = 768, num_patches: int = 1024, num_actions: int = 5):
super().__init__()
self.embedding_dim = embedding_dim
self.num_patches = num_patches
self.num_actions = num_actions
# ACTOR token
self.actor_token = nn.Parameter(torch.randn(1, 1, embedding_dim))
# Cross-attention mechanism
self.actor_attention = nn.MultiheadAttention(embedding_dim, num_heads=8, batch_first=True)
# Action proposal generation
self.action_mlp = nn.Sequential(
nn.Linear(embedding_dim, 512),
nn.ReLU(),
nn.Linear(512, num_patches)
)
def forward(self, patch_embeddings: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
"""
Generate action candidates via attention alignment.
Args:
patch_embeddings: (batch_size, num_patches, embedding_dim)
Returns:
action_logits: (batch_size, num_patches) - logits for each patch
actor_embeddings: (batch_size, embedding_dim) - refined actor token
"""
batch_size = patch_embeddings.shape[0]
# Expand ACTOR token for batch
actor = self.actor_token.expand(batch_size, -1, -1)
# Cross-attention: ACTOR attends to patches
actor_attended, _ = self.actor_attention(actor, patch_embeddings, patch_embeddings)
# Generate action logits for each patch
action_logits = self.action_mlp(actor_attended.squeeze(1))
return action_logits, actor_attended.squeeze(1)
class GroundingVerifier(nn.Module):
"""Verify and rank action candidates."""
def __init__(self, embedding_dim: int = 768):
super().__init__()
self.scoring_network = nn.Sequential(
nn.Linear(embedding_dim * 2, 256),
nn.ReLU(),
nn.Dropout(0.1),
nn.Linear(256, 1),
nn.Sigmoid()
)
def verify_candidates(self, actor_embedding: torch.Tensor, patch_embeddings: torch.Tensor,
action_logits: torch.Tensor, top_k: int = 3) -> Tuple[torch.Tensor, List[int]]:
"""
Verify and rank action candidates.
Args:
actor_embedding: (batch_size, embedding_dim)
patch_embeddings: (batch_size, num_patches, embedding_dim)
action_logits: (batch_size, num_patches)
top_k: Number of top candidates to return
Returns:
verified_scores: (batch_size, top_k) - confidence scores
selected_patches: (batch_size, top_k) - patch indices
"""
batch_size, num_patches, _ = patch_embeddings.shape
# Get top-k candidates by logits
top_logits, top_indices = torch.topk(action_logits, k=top_k, dim=1)
# Score candidates using verifier
verified_scores = []
for b in range(batch_size):
scores = []
for idx in top_indices[b]:
patch_emb = patch_embeddings[b, idx, :]
combined = torch.cat([actor_embedding[b], patch_emb])
score = self.scoring_network(combined)
scores.append(score.item())
verified_scores.append(scores)
verified_scores = torch.tensor(verified_scores)
return verified_scores, top_indices
def select_action(self, verified_scores: torch.Tensor, patch_indices: torch.Tensor) -> torch.Tensor:
"""Select best action from verified candidates."""
best_indices = torch.argmax(verified_scores, dim=1)
selected_patches = torch.gather(patch_indices, 1, best_indices.unsqueeze(1))
return selected_patches
class GUIActor(nn.Module):
"""Coordinate-free GUI action grounding."""
def __init__(self, vit_model, embedding_dim: int = 768, num_patches: int = 1024):
super().__init__()
self.vit = vit_model
self.action_head = ActionHead(embedding_dim, num_patches)
self.verifier = GroundingVerifier(embedding_dim)
self.patch_extractor = PatchExtractor()
def ground_action(self, image: torch.Tensor) -> Tuple[int, float]:
"""
Ground an action in the GUI without explicit coordinates.
Args:
image: Screenshot tensor (batch_size, 3, H, W)
Returns:
patch_id: Selected patch index
confidence: Confidence score for the action
"""
# Extract patch embeddings from frozen ViT backbone
with torch.no_grad():
patch_embeddings = self.vit.get_patch_embeddings(image)
# Generate action candidates
action_logits, actor_embedding = self.action_head(patch_embeddings)
# Verify and select best action
verified_scores, top_indices = self.verifier.verify_candidates(
actor_embedding, patch_embeddings, action_logits, top_k=5
)
selected_patch = self.verifier.select_action(verified_scores, top_indices)
return selected_patch.squeeze(), verified_scores.max().item()
def forward(self, images: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
"""Forward pass for batch processing."""
batch_size = images.shape[0]
# Extract patch embeddings
with torch.no_grad():
patch_embeddings = self.vit.get_patch_embeddings(images)
# Generate action candidates
action_logits, actor_embedding = self.action_head(patch_embeddings)
# Verify candidates
verified_scores, top_indices = self.verifier.verify_candidates(
actor_embedding, patch_embeddings, action_logits, top_k=5
)
selected_patches = self.verifier.select_action(verified_scores, top_indices)
return selected_patches, verified_scores
def patch_to_coordinates(patch_id: int, patch_size: int = 16, image_size: int = 512) -> Tuple[int, int]:
"""Convert patch ID back to approximate coordinates."""
patches_per_side = image_size // patch_size
row = (patch_id // patches_per_side) * patch_size + patch_size // 2
col = (patch_id % patches_per_side) * patch_size + patch_size // 2
return col, row
Practical Guidance
Fine-tuning strategy:
- Frozen backbone: Keep VLM frozen during training; only update action head (~100M params)
- Learning rate: 1e-3 to 1e-4 for minimal backbone disturbance
- Batch size: 32-64 sufficient given limited trainable parameters
- Epochs: 3-10 epochs typically converge; monitor validation performance
Verifier configuration:
- Top-k candidates: 3-5 candidates; too few limits options, too many adds noise
- Scoring network depth: 2-3 layers; balance expressiveness with overfitting risk
- Confidence threshold: 0.7+ for filtering low-confidence actions
When to use:
- GUI automation and interaction tasks
- Mobile app testing and automation
- Web scraping requiring visual understanding
- Interactive systems where coordinate precision is difficult
- Transfer learning to new interfaces without coordinate retraining
When NOT to use:
- Pixel-level precision required (e.g., precise drawing tools)
- Dynamic interfaces with rapidly changing layouts
- Tasks where coordinate output is explicitly needed downstream
- Real-time systems where attention computation overhead is prohibitive
Common pitfalls:
- Frozen VLM limitation: Backbone features may not be optimal for action grounding; consider light fine-tuning
- Patch resolution tradeoff: Larger patches lose fine-grained localization; smaller patches increase computation
- Verifier collapse: Scoring network may converge to trivial solutions; use diverse training data
- Generalization: Models may overfit to specific UI layouts; validate on diverse interfaces
- Confidence calibration: Verified scores may not reflect true reliability; apply temperature scaling
Reference
GUI-Actor achieves 44.6 on ScreenSpot-Pro with Qwen2.5-VL, surpassing larger models (UI-TARS-72B at 38.1). The approach improves generalization to unseen resolutions and demonstrates effective capability transfer from frozen VLMs.
Original paper: "GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents" (arxiv.org/abs/2506.03143)