Screenspot Pro Eval

screenspot-pro-eval

ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use — Li et al. (2025) (arXiv:2504.07981, 2025)

What this evaluates

This benchmark evaluates a model's ability to perform GUI grounding in professional, high-resolution desktop environments. It probes whether vision-language models can accurately locate specific UI elements (both text and icons) based on natural language instructions, highlighting challenges with small targets and complex interfaces.

Datasets

• ScreenSpot-Pro — total ?; splits: Development (-1), Creative (-1), CAD (-1), Scientific (-1), Office (-1), OS (-1)

Metrics

• accuracy (center-point) (primary) — range: [0, 1] | percent
  • Accuracy is calculated as the fraction of test instances where the center point of the predicted bounding box falls within the annotated ground truth bounding box.

Input / output format

Input: High-resolution desktop screenshot image paired with a natural language instruction specifying the target UI element.

Output: Bounding box coordinates (x_min, y_min, x_max, y_max) representing the predicted location of the target element.

Scoring recipe

correct = 0
for pred_box, gt_box in zip(predictions, ground_truths):
    pred_center = ((pred_box.x_min + pred_box.x_max) / 2, (pred_box.y_min + pred_box.y_max) / 2)
    if gt_box.x_min <= pred_center[0] <= gt_box.x_max and gt_box.y_min <= pred_center[1] <= gt_box.y_max:
        correct += 1
accuracy = correct / len(predictions)

Common pitfalls

• Icon targets are significantly harder to ground than text targets due to lack of textual cues and domain-specific meanings.
• Direct grounding performance is highly sensitive to crop size; too small crops lose context while too large crops exceed model processing capacity.
• Models perform poorly on direct grounding (e.g., 18.9% accuracy), requiring iterative search or refinement strategies to achieve competitive results.

Evidence (verbatim from paper)

With ScreenSpot-Pro, we rigorously evaluate the correctness whether the model’s predictions fall into the annotated ground truth boxes. For models inferencing boxes, we consider the center point of the generated box as the prediction. OS-Atlas-7B leads the performance with an accuracy of 18.9%, closely followed by UGround and AriaUI.

Citation

@misc{li2025screenspotpro,
  title={ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2504.07981}
}

• arXiv: 2504.07981