Mobibench Eval

mobibench-eval

Modular and Multi-Path-Aware Offline Benchmarking for Mobile GUI Agents — Im et al. (2025) (arXiv:2512.12634, 2025)

What this evaluates

Evaluates mobile GUI agents' ability to complete tasks on mobile interfaces by measuring task success rate across diverse, multi-path trajectories. It also enables modular component analysis (screen parsing, history generation, inference style, reflection) to identify performance bottlenecks and optimal configurations for different foundation models.

Datasets

• MobiBench — total 105; splits: test (105); repo https://github.com/fclab-skku/Mobi-Bench

Metrics

• Task Success Rate (TSR) (primary) — range: percent
  • Percentage of tasks completed successfully. Calculated as (number of successfully completed tasks / total number of tasks) * 100. Ground truth is validated by human judges with 93.01% inter-annotator agreement.
• Action Accuracy (A.Acc) — range: [0, 1]
  • Percentage of individual steps/actions predicted correctly by the agent at each turn.
• Cost — range: other
  • Monetary cost per task in USD, calculated based on the input/output token pricing of the underlying foundation model.

Input / output format

Input: Task instruction (natural language goal) + current mobile UI state (screenshot and/or accessibility tree) + conversation history.

Output: Sequence of GUI actions (e.g., click, type, scroll, back) or a completion signal.

Scoring recipe

def compute_tsr(predictions, valid_paths):
    correct = 0
    for pred_trace, paths in zip(predictions, valid_paths):
        if any(trace_matches(pred_trace, path) for path in paths):
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Single-path offline benchmarks systematically penalize agents for choosing equally valid alternative paths, leading to artificially low success rates.
• Naive self-reflection often backfires due to false alarms, incorrectly modifying correct actions and degrading overall performance.
• Assuming ReAct prompting or Set-of-Mark are universally optimal can lead to suboptimal configurations; effectiveness heavily depends on the underlying model size and type.

Evidence (verbatim from paper)

Table 4 reports the Task Success Rate (TSR) of m3a under four different evaluation settings. ... For ground truth, three human judges manually validated the agent’s full execution traces (inter-annotator agreement: 93.01%).

Citation

@misc{im2025mobibench,
  title={Modular and Multi-Path-Aware Offline Benchmarking for Mobile GUI Agents},
  author={Im et al. (2025)},
  year={2025},
  note={arXiv:2512.12634}
}

• arXiv: 2512.12634