qhjqhj00 showdown-clicks-eval
WebSight: A Vision-First Architecture for Robust Web Agents — Bhathal et al. (2025) (arXiv:2508.16987, 2025)
What this evaluates
Evaluates a vision-language model's ability to accurately predict the correct UI element to click based on a screenshot and a task instruction. It isolates low-level visual grounding and interaction skills in ambiguous or icon-heavy interfaces.
Datasets
- Showdown-Clicks — total 5679; splits: test (5679), dev (557)
Metrics
Top-1 Accuracy(primary) — range: percent- Percentage of test cases where the model's predicted click location exactly matches the ground-truth human click. Calculated as (correct predictions / total examples) * 100.
Input / output format
Input: A screenshot of a macOS desktop/UI element and a natural language task instruction.
Output: A single click coordinate (x, y) or UI element identifier corresponding to the target.
Scoring recipe
correct = 0
for pred, gold in zip(predictions, golds):
if pred == gold:
correct += 1
accuracy = (correct / len(golds)) * 100
Common pitfalls
- Ambiguous UI contexts where multiple elements could satisfy the instruction.
- Difficulty distinguishing interactive icons from static visual elements or text labels.
- Model may output actions like scrolling or text input when a direct click is required.
Evidence (verbatim from paper)
The showdown-clicks track contains 5679 human-collected left-click events on macOS, with a public dev subset of 557 examples. ... WEBSIGHT-7B attains 58.84% accuracy, achieving higher accuracy than VLMs with almost 10x more parameters. Table 1. Top-1 Accuracy on the Showdown/Clicks Benchmark [59]
Citation
@misc{bhathal2025websight,
title={WebSight: A Vision-First Architecture for Robust Web Agents},
author={Bhathal et al. (2025)},
year={2025},
note={arXiv:2508.16987}
}
- arXiv: 2508.16987