# Mmbench Gui Eval

> mmbench-gui-eval

- Skill: `qhjqhj00/mmbench-gui-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/mmbench-gui-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mmbench-gui-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/mmbench-gui-eval

---


# mmbench-gui-eval

> MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents — Xuehui Wang et al. (arXiv:2507.19478, 2025)

## What this evaluates

Evaluates GUI agents across four hierarchical levels: content understanding, element grounding, single-app task automation, and multi-app task collaboration. It probes visual grounding, cross-platform generalization, and long-horizon planning capabilities while measuring both task success and step efficiency.

## Datasets

- **MMBench-GUI** — total ?; splits: test (-1); repo https://github.com/open-compass/MMBench-GUI

## Metrics

- `exact-match accuracy` **(primary)** — range: percent
  - Returns 1 if the model's parsed output exactly matches the gold answer, else 0. Scores are aggregated as a weighted average across platforms based on task proportions.
- `success_rate (SR)` — range: percent
  - Returns 1 if the agent's final GUI state matches the predefined success criteria, else 0. Averaged across all tasks and platforms.
- `EQA` — range: other
  - Efficiency–Quality-Aware metric that jointly measures task accuracy and step efficiency. It rewards successful task completion while penalizing redundant or inefficient action sequences.

## Input / output format

**Input**: GUI screenshot paired with a task description or instruction/question. Auxiliary artifacts such as accessibility (A11y) trees and Set-of-Marks (SoM) data are deliberately omitted.

**Output**: L1/L2: Natural language answer or element identifier. L3/L4: Sequence of GUI actions/steps leading to a final state.

## Scoring recipe

```python
def score_l1_l2(pred, gold):
    parsed = parse_with_regex(pred)
    return 1.0 if parsed == gold else 0.0

def score_l3_l4(pred_actions, gold_state, max_steps=50):
    success = check_final_state(pred_actions, gold_state)
    if success:
        eqa = 1.0 / len(pred_actions)  # Penalizes longer sequences
        return {"SR": 1.0, "EQA": eqa}
    return {"SR": 0.0, "EQA": 0.0}

# Aggregate across tasks/platforms using weighted averages
```

## Common pitfalls

- Output parsing for L1/L2 is highly sensitive to model instruction-following variability, often requiring custom regex-based parsers to extract valid answers.
- L3/L4 evaluation only checks the final state against success criteria, ignoring intermediate steps and masking inefficient or redundant action sequences.
- Platform scores must be weighted by task proportion before aggregation; simple averaging skews results due to uneven task distribution across OS/browser environments.

## Evidence (verbatim from paper)

> Model outputs were assessed using exact-match evaluation protocol, analogous to standard practices in grounding and QA tasks. However, given the variability in instruction-following abilities across different models, for example, the QA tasks in L1, we observed that some model outputs could not be reliably parsed. To address this, we implemented a hybrid parsing mechanism based on multiple regular expressions to robustly extract valid answers.

## Citation

```bibtex
@misc{wang2025mmbenchgui,
  title={MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents},
  author={Xuehui Wang et al.},
  year={2025},
  note={arXiv:2507.19478}
}
```

- arXiv: 2507.19478