Screenspot Eval

screenspot-eval

SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents — Cheng et al. (2024) (arXiv:2401.10935, 2024)

What this evaluates

Evaluates a model's ability to locate specific UI elements on screenshots based on natural language instructions. It measures how accurately the model can predict click coordinates that align with ground-truth bounding boxes across mobile, desktop, and web platforms.

Datasets

• ScreenSpot — total ?; splits: test (-1); repo https://github.com/njucckevin/SeeClick

Metrics

• click accuracy (primary) — range: percent
  • Proportion of test samples where the model's predicted location (point or center of predicted bounding box) falls within the ground-truth element bounding box.

Input / output format

Input: Screenshot of a GUI interface paired with a natural language instruction describing the target element.

Output: A predicted 2D coordinate (point) or bounding box representing the target element location.

Scoring recipe

correct = 0
for pred_box, gt_box in zip(predictions, ground_truth):
    pred_pt = center(pred_box) if is_box(pred_box) else pred_box
    if point_in_box(pred_pt, gt_box):
        correct += 1
return correct / len(predictions) * 100

Common pitfalls

• Models often struggle with non-text elements like icons/widgets, leading to artificially low scores if only text grounding is evaluated.
• Using the center of a predicted bounding box vs. a direct point prediction can yield different results; the protocol explicitly allows both but requires consistent handling.

Evidence (verbatim from paper)

Considering that GUI agents require clicking on the correct position, we calculate the click accuracy as the metric, defined as the proportion of test samples where the model’ predicted location falls in the groundtruth element bounding box. We take the models’ generated point or the center of the generated bounding box as the predicted click location.

Citation

@misc{cheng2024seeclick,
  title={SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents},
  author={Cheng et al. (2024)},
  year={2024},
  note={arXiv:2401.10935}
}

• arXiv: 2401.10935