Mmbench Gui Eval

mmbench-gui-eval

MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents — Xuehui Wang et al. (arXiv:2507.19478, 2025)

What this evaluates

Evaluates GUI agents across four hierarchical levels: content understanding, element grounding, single-app task automation, and multi-app task collaboration. It probes visual grounding, cross-platform generalization, and long-horizon planning capabilities while measuring both task success and step efficiency.

Datasets

• MMBench-GUI — total ?; splits: test (-1); repo https://github.com/open-compass/MMBench-GUI

Metrics

• exact-match accuracy (primary) — range: percent
  • Returns 1 if the model's parsed output exactly matches the gold answer, else 0. Scores are aggregated as a weighted average across platforms based on task proportions.
• success_rate (SR) — range: percent
  • Returns 1 if the agent's final GUI state matches the predefined success criteria, else 0. Averaged across all tasks and platforms.
• EQA — range: other
  • Efficiency–Quality-Aware metric that jointly measures task accuracy and step efficiency. It rewards successful task completion while penalizing redundant or inefficient action sequences.

Input / output format

Input: GUI screenshot paired with a task description or instruction/question. Auxiliary artifacts such as accessibility (A11y) trees and Set-of-Marks (SoM) data are deliberately omitted.

Output: L1/L2: Natural language answer or element identifier. L3/L4: Sequence of GUI actions/steps leading to a final state.

Scoring recipe

def score_l1_l2(pred, gold):
    parsed = parse_with_regex(pred)
    return 1.0 if parsed == gold else 0.0

def score_l3_l4(pred_actions, gold_state, max_steps=50):
    success = check_final_state(pred_actions, gold_state)
    if success:
        eqa = 1.0 / len(pred_actions)  # Penalizes longer sequences
        return {"SR": 1.0, "EQA": eqa}
    return {"SR": 0.0, "EQA": 0.0}

# Aggregate across tasks/platforms using weighted averages

Common pitfalls

• Output parsing for L1/L2 is highly sensitive to model instruction-following variability, often requiring custom regex-based parsers to extract valid answers.
• L3/L4 evaluation only checks the final state against success criteria, ignoring intermediate steps and masking inefficient or redundant action sequences.
• Platform scores must be weighted by task proportion before aggregation; simple averaging skews results due to uneven task distribution across OS/browser environments.

Evidence (verbatim from paper)

Model outputs were assessed using exact-match evaluation protocol, analogous to standard practices in grounding and QA tasks. However, given the variability in instruction-following abilities across different models, for example, the QA tasks in L1, we observed that some model outputs could not be reliably parsed. To address this, we implemented a hybrid parsing mechanism based on multiple regular expressions to robustly extract valid answers.

Citation

@misc{wang2025mmbenchgui,
  title={MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents},
  author={Xuehui Wang et al.},
  year={2025},
  note={arXiv:2507.19478}
}

• arXiv: 2507.19478