Screen Spot Grounding Eval

screen-spot-grounding-eval

OS-ATLAS: A Foundation Action Model for Generalist GUI Agents — Zhiyong Wu et al. (2024) (arXiv:2410.23218, 2024)

What this evaluates

Evaluates a model's ability to locate specific GUI elements from a screenshot given a text instruction. It measures both coarse localization accuracy and fine-grained bounding box overlap across desktop, mobile, and web platforms.

Datasets

• ScreenSpot — total ?; splits: test (-1)

Metrics

• grounding accuracy (primary) — range: [0, 1]
  • Fraction of test instances where the predicted bounding box falls entirely within the ground truth bounding box.
• IoU — range: [0, 1]
  • Intersection over Union: area of overlap between predicted and ground truth bounding boxes divided by their union area.

Input / output format

Input: GUI screenshot image paired with a natural language instruction (optionally pre-processed by a planner model in the Grounding Mode Setting).

Output: Bounding box coordinates (x_min, y_min, x_max, y_max) or a point coordinate indicating the target GUI element.

Scoring recipe

def compute_metrics(predictions, golds):
    accs, ious = [], []
    for pred, gold in zip(predictions, golds):
        ix1, iy1 = max(pred[0], gold[0]), max(pred[1], gold[1])
        ix2, iy2 = min(pred[2], gold[2]), min(pred[3], gold[3])
        inter = max(0, ix2 - ix1) * max(0, iy2 - iy1)
        union = (pred[2]-pred[0])*(pred[3]-pred[1]) + (gold[2]-gold[0])*(gold[3]-gold[1]) - inter
        ious.append(inter / union if union > 0 else 0.0)
        accs.append(1.0 if (pred[0]>=gold[0] and pred[1]>=gold[1] and pred[2]<=gold[2] and pred[3]<=gold[3]) else 0.0)
    return {'grounding_accuracy': sum(accs)/len(accs), 'iou': sum(ious)/len(ious)}

Common pitfalls

• ScreenSpot contains ~11.32% annotation errors; using the raw dataset without correction (ScreenSpot-V2) may yield inaccurate baseline comparisons.
• Grounding accuracy is a coarse metric that ignores fine-grained localization errors; IoU should be reported alongside it to capture precise bounding box overlap.
• Performance varies significantly between the 'Standard Setting' (direct instruction) and 'Grounding Mode Setting' (planner-refined instruction); results are not directly comparable across settings.

Evidence (verbatim from paper)

We follow previous practices by using grounding accuracy on ScreenSpot, where a prediction is considered correct if the predicted location falls within the ground truth element’s bounding box. However, this metric does not capture more fine-grained grounding errors. Therefore, we also use Intersection over Union (IoU), a widely used metric for measuring localization accuracy in object detection. IoU quantifies the overlap between the predicted bounding box and the ground truth bounding box.

Citation

@misc{wu2024osatlas,
  title={OS-ATLAS: A Foundation Action Model for Generalist GUI Agents},
  author={Zhiyong Wu et al. (2024)},
  year={2024},
  note={arXiv:2410.23218}
}

• arXiv: 2410.23218