qhjqhj00 mind2web-live-eval
Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills — Xie et al. (2025) (arXiv:2506.10387, 2025)
What this evaluates
Evaluates a GUI agent's ability to perform web browsing tasks using either HTML tree or image inputs. It measures the agent's capacity to navigate websites and complete user intents across diverse web interfaces.
Datasets
- Mind2Web-Live — total ?; splits: test (-1)
Metrics
task success rate(primary) — range: percent- Percentage of tasks completed successfully out of the total number of tasks evaluated.
micro-completion rate— range: percent- Measures the proportion of key nodes completed across all tasks.
Input / output format
Input: Web page observations provided as either HTML tree or screenshots.
Output: Action selection (e.g., click, type, scroll) to interact with the web interface.
Scoring recipe
success = 1 if task goal is met, else 0.
SR = mean(success) * 100.
CR = (total key nodes completed / total key nodes) * 100.
Common pitfalls
- Micro-completion rate aggregates across tasks, which can mask per-task failure patterns and overstate agent capability.
- HTML tree vs image input modes significantly affect baseline performance and should be matched carefully when comparing methods.
Evidence (verbatim from paper)
For Web, we employ Mind2Web-Live as the evaluation benchmark. We use task success rate and micro-completion rate (measures the proportion of key nodes completed in all tasks) as evaluation metrics.
Citation
@misc{xie2025mirage1,
title={Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills},
author={Xie et al. (2025)},
year={2025},
note={arXiv:2506.10387}
}
- arXiv: 2506.10387