qhjqhj00 screen-spot-eval
ZonUI-3B: A Lightweight Vision-Language Model for Cross-Resolution GUI Grounding — Hsieh et al. (2025) (arXiv:2506.23491, 2025)
What this evaluates
This evaluation probes a vision-language model's ability to localize specific UI elements within graphical user interfaces based on natural language instructions. It tests precise coordinate prediction and cross-resolution generalization across mobile, desktop, and web platforms.
Datasets
- ScreenSpot — total 1272; splits: test (1272)
- ScreenSpot-v2 — total 1272; splits: test (1272)
- ScreenSpot-Pro — total 1272; splits: test (1272)
Metrics
accuracy(primary) — range: percent- Success is defined by whether the predicted coordinates fall within the annotated bounding box. Accuracy is calculated as the percentage of correctly grounded tasks out of the total instances.
Input / output format
Input: A GUI screenshot image paired with a natural language instruction describing the target UI element to locate.
Output: Predicted bounding box coordinates (e.g., [x_min, y_min, x_max, y_max]) for the target element.
Scoring recipe
def compute_accuracy(predictions, gold_boxes):
correct = 0
for pred_box, gold_box in zip(predictions, gold_boxes):
if is_within(pred_box, gold_box):
correct += 1
return (correct / len(predictions)) * 100
Common pitfalls
- Confusing ScreenSpot with ScreenSpot-v2, which contains corrected annotations and clarified instructions, leading to different baseline scores.
- Assuming high accuracy on standard-resolution screenshots implies robustness on ScreenSpot-Pro, which features dense professional software layouts with significantly smaller UI targets.
Evidence (verbatim from paper)
On ScreenSpot, ZonUI-3B achieves an accuracy of 84.9%, and on the cleaned ScreenSpot-v2, it reaches 86.4%, setting a new benchmark among all sub-4B models.
Citation
@misc{hsieh2025zonui,
title={ZonUI-3B: A Lightweight Vision-Language Model for Cross-Resolution GUI Grounding},
author={Hsieh et al. (2025)},
year={2025},
note={arXiv:2506.23491}
}
- arXiv: 2506.23491