Mobile Gui Agent Eval

mobile-gui-agent-eval

UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents — Han Xiao et al. (arXiv:2505.21496, 2025)

What this evaluates

Evaluates the capability of multimodal large language models to act as autonomous mobile GUI agents. It probes their ability to plan tasks, predict correct interaction types, accurately ground UI elements, and successfully complete complex, multi-step workflows in both static and dynamic Android environments.

Datasets

• AndroidControl — total ?; splits: test (-1)
• AndroidLab — total 138; splits: test (138)
• Android Agent Arena (A3) — total 201; splits: test (201)

Metrics

• Success Rate (SR) (primary) — range: percent
  • Measures the percentage of successfully completed tasks or actions. Calculated as the number of successful tasks divided by the total number of tasks.
• Type Accuracy — range: percent
  • Assesses the agent’s ability to correctly predict action types such as click, long-press, or scroll. Calculated as the number of correct action type predictions divided by total predictions.
• Grounding Accuracy — range: percent
  • Evaluates the precision in locating and interacting with the correct UI elements. Calculated as the number of correctly grounded interactions divided by total interactions.
• Essential State Achievement Rate (ESAR) — range: percent
  • Measures the rate at which agents achieve critical intermediate states required for task completion, evaluated via function-based or LLM-based essential state checks.

Input / output format

Input: Current and historical screenshots, task instructions/goals, and action history. In dynamic settings, the model receives live emulator state updates after each action.

Output: Predicted action type (e.g., click, scroll, long-press) and UI element grounding coordinates/identifiers.

Scoring recipe

def compute_metrics(predictions, golds):
    type_correct = sum(1 for p, g in zip(predictions, golds) if p.action_type == g.action_type)
    grounding_correct = sum(1 for p, g in zip(predictions, golds) if p.grounding == g.grounding)
    task_success = sum(1 for p, g in zip(predictions, golds) if p.task_completed == g.task_completed)
    n = len(predictions)
    return {
        'Type Accuracy': type_correct / n,
        'Grounding Accuracy': grounding_correct / n,
        'Success Rate': task_success / n
    }

Common pitfalls

• Static vs. dynamic evaluation protocols differ significantly; static benchmarks only compare predicted actions against ground truth without executing them, while dynamic benchmarks run actions in an emulator, making success dependent on state transitions.
• Grounding Accuracy is frequently reported as 0.0 for proprietary models (e.g., Claude, GPT-4o) in Table 2, likely because they do not output explicit UI element coordinates/IDs required for this metric.
• Task complexity categories (easy/medium/hard) are strictly defined by step count (<5, 5-10, >10) and must be used when reporting stratified performance.

Evidence (verbatim from paper)

For both settings, we employ three primary metrics: (1) Success Rate (SR), measuring the percentage of successfully completed actions; (2) Type Accuracy, assessing the agent’s ability to correctly predict action types (e.g., click, long-press, scroll); and (3) Grounding Accuracy, evaluating the agent’s precision in locating and interacting with the correct UI elements.

Citation

@misc{xiao2025uigenie,
  title={UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents},
  author={Han Xiao et al.},
  year={2025},
  note={arXiv:2505.21496}
}

• arXiv: 2505.21496