Mobile Bench Eval

mobile-bench-eval

Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents — Shihan Deng et al. (2024) (arXiv:2407.00993, 2024)

What this evaluates

Evaluates LLM-based mobile agents on real-world task execution across single-app and multi-app scenarios. It probes the agent's ability to plan, navigate UIs, call APIs, and collaborate across applications to complete user-defined goals.

Datasets

• Mobile-Bench — total ?; splits: SAST (-1), SAMT (-1), MAMT (-1); repo https://github.com/XiaoMi/MobileBench

Metrics

• PassRate (primary) — range: percent
  • Percentage of tasks successfully completed by the agent within the step limit.
• CheckPointl1 — range: percent
  • Coverage of critical planning milestones at level 1, measuring whether the agent reaches key decision points during execution.
• CheckPointl2 — range: percent
  • Coverage of critical planning milestones at level 2, measuring deeper planning and multi-app transition milestones.
• Average #Steps — range: other
  • Mean number of execution steps (API calls or UI operations) taken per task.

Input / output format

Input: Task description, target APP list, available API list, maximum step limit, initial thought, and current UI state (HTML/XML) retrieved via Appium.

Output: Iteratively outputs either an API call or a UI operation, followed by a thought and a finish flag indicating task completion.

Scoring recipe

def compute_metrics(predictions, gold):
    n = len(predictions)
    pass_rate = sum(1 for p in predictions if p['finish'] and verify(p)) / n * 100
    cp1 = sum(1 for m in gold['milestones_l1'] if m['reached']) / len(gold['milestones_l1']) * 100
    cp2 = sum(1 for m in gold['milestones_l2'] if m['reached']) / len(gold['milestones_l2']) * 100
    avg_steps = sum(len(p['actions']) for p in predictions) / n
    return {'PassRate': pass_rate, 'CheckPointl1': cp1, 'CheckPointl2': cp2, 'Average #Steps': avg_steps}

Common pitfalls

• Models prematurely terminate tasks due to misjudging completion state, inflating PassRate while missing goals.
• Context length limits force action history compression, degrading performance on longer MAMT tasks.
• UI operations (scrolling, clicking) are inefficient compared to APIs, causing high step counts and navigation failures.

Evidence (verbatim from paper)

Tasks are categorized into SAST, SAMT, and MAMT to assess varying levels of planning and reasoning, with real user queries and multi-APP collaboration scenarios reflecting real-world complexity. A new evaluation metric, CheckPoint, is proposed to objectively verify whether agents reach critical planning milestones during task execution, addressing limitations in existing benchmarks that rely solely on final outcomes or simulated environments.

Citation

@misc{deng2024mobilebench,
  title={Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents},
  author={Shihan Deng et al. (2024)},
  year={2024},
  note={arXiv:2407.00993}
}

• arXiv: 2407.00993