qhjqhj00 gui-grounding-agent-eval
GTA1: GUI Test-time Scaling Agent — Yan Yang et al. (2025) (arXiv:2507.05791, 2025)
What this evaluates
Evaluates a GUI agent's ability to precisely locate UI elements (grounding) and execute multi-step tasks in real-world desktop/web environments. It probes spatial reasoning, text/icon matching, and long-horizon planning robustness without lookahead.
Datasets
- ScreenSpot-Pro — total ?; splits: test (-1)
- ScreenSpot-V2 — total ?; splits: test (-1)
- OSWorld-G — total ?; splits: test (-1)
- OSWorld — total ?; splits: test (-1)
- WindowsAgentArena — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of test instances where the predicted click coordinate falls within the target UI element's bounding box or matches the ground truth coordinate exactly.
task success rate— range: percent- Percentage of multi-step agent tasks that are fully completed successfully in the dynamic environment, judged by a separate model (o3).
Input / output format
Input: GUI screen capture (image), task instruction text, and optionally a chain-of-thought reasoning trace.
Output: For grounding: a single (x, y) coordinate. For agent tasks: a sequence of discrete actions (e.g., click, type, scroll) executed step-by-step.
Scoring recipe
def score_grounding(pred, gold):
correct = sum(1 for p, g in zip(pred, gold) if is_within_target(p, g))
return correct / len(gold) * 100
def score_agent(pred_actions, gt_tasks, judge):
success = sum(1 for p, t in zip(pred_actions, gt_tasks) if judge.evaluate(t, p) == 'success')
return success / len(gt_tasks) * 100
Common pitfalls
- Grounding accuracy relies on a specific coordinate-matching threshold; bounding box regression metrics are explicitly avoided in favor of click rewards.
- Task success rate is heavily dependent on the external planner (o3) and judge model, so it measures the full agent pipeline rather than the grounding model alone.
- OSWorld-Verified is a distinct, updated variant of OSWorld; using the original split will yield different results.
Evidence (verbatim from paper)
We evaluate our method on two sets of benchmarks: i) GUI Grounding, where we use ScreenSpot-Pro [3], ScreenSpot-V2 [4, 5], and OSWorld-G [11] datasets, evaluating by the metric of accuracy; ii) Agent Task Execution, where we use OSWorld [11] and WindowsAgentArena [33] benchmarks, measuring performance by task success rate.
Citation
@misc{yang2025gta1,
title={GTA1: GUI Test-time Scaling Agent},
author={Yan Yang et al. (2025)},
year={2025},
note={arXiv:2507.05791}
}
- arXiv: 2507.05791