Screen Parsing Eval

screen-parsing-eval

GUI-360$^\circ$: A Comprehensive Dataset and Benchmark for Computer-Using Agents — Jian Mu et al. (2025) (arXiv:2511.04307, 2025)

What this evaluates

Evaluates a model's ability to detect, localize, and semantically label all interactable UI elements on a clean screenshot. It probes fine-grained spatial reasoning, handling of dense layouts, and UI semantics understanding.

Datasets

• GUI-360°-Bench — total ?; splits: train (-1), test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall computed per image via greedy bipartite matching (IoU > 0.5), then macro-averaged across all images.
• mean IoU — range: [0, 1]
  • Average intersection-over-union score for matched element pairs per image, macro-averaged across the benchmark.
• semantic name accuracy — range: [0, 1]
  • Cosine similarity between sentence embeddings of predicted and ground-truth element names for matched pairs, macro-averaged.

Input / output format

Input: Clean screenshot of the GUI state.

Output: Set of interactable UI elements, each consisting of a semantic name (string) and a bounding box (2D coordinates).

Scoring recipe

def compute_metrics(predictions, ground_truths):
    matched_pairs = []
    for pred, gt in zip(predictions, ground_truths):
        pairs = greedy_match(pred, gt, iou_threshold=0.5)
        matched_pairs.append(pairs)
    precisions, recalls, ious, sims = [], [], [], []
    for pairs in matched_pairs:
        if not pairs: continue
        precisions.append(len(pairs)/len(pred))
        recalls.append(len(pairs)/len(gt))
        ious.append(mean_iou(pairs))
        sims.append(mean_text_sim(pairs))
    return macro_avg(precisions), macro_avg(recalls), macro_avg(ious), macro_avg(sims)

Common pitfalls

• General-purpose VLMs exhibit very low recall, missing many elements due to dense layouts and occlusions.
• Metrics are computed per-image then macro-averaged; failing to average per-image before aggregation will skew results toward larger images.

Evidence (verbatim from paper)

We measure parsing quality along three complementary axes: (i) element detection accuracy (precision / recall / F1), (ii) localization quality (mean IoU on matched pairs), and (iii) semantic name accuracy (average text embedding similarity on matched pairs).

Citation

@misc{mu2025gui360,
  title={GUI-360$^\circ$: A Comprehensive Dataset and Benchmark for Computer-Using Agents},
  author={Jian Mu et al. (2025)},
  year={2025},
  note={arXiv:2511.04307}
}

• arXiv: 2511.04307