qhjqhj00 androidlh-eval
Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills — Xie et al. (2025) (arXiv:2506.10387, 2025)
What this evaluates
Evaluates a GUI agent's ability to perform long-horizon, multi-app tasks in a mobile environment. It probes the agent's planning and skill-retrieval capabilities across complex, real-world application scenarios.
Datasets
- AndroidLH — total 30; splits: test (30)
Metrics
task success rate(primary) — range: percent- Percentage of long-horizon tasks completed successfully.
completion rate— range: percent- Proportion of sub-goals or steps completed successfully across tasks.
Input / output format
Input: Screen screenshots of the Android emulator environment for multi-app, long-horizon tasks.
Output: Action selection to navigate and interact across multiple applications.
Scoring recipe
success = 1 if task goal is met according to system state-based verification, else 0.
SR = mean(success) * 100.
CR = mean(completion proportion) * 100.
Common pitfalls
- Tasks are synthetically generated by GPT-4o, which may not perfectly reflect real-world complexity or edge cases.
- System state-based verification requires accurate state tracking across app boundaries, which can be brittle if the emulator state diverges from the agent's perception.
Evidence (verbatim from paper)
Specifically, Mirage-1 exhibits significant performance gains with mean improvements of 53.3% in completion rate and 56.1% in success rate compared to the baseline agents.
Citation
@misc{xie2025mirage1,
title={Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills},
author={Xie et al. (2025)},
year={2025},
note={arXiv:2506.10387}
}
- arXiv: 2506.10387