# Mobibench Eval

> Evaluates mobile GUI agents' ability to complete tasks on mobile interfaces by measuring task success rate across diverse, multi-path trajectories. It also enables modular component analysis (screen parsing, history generation, inference style, reflection) to identify performance bottlenecks and optimal configurations for different foundation models. Use when the user wants to benchmark on MobiBench, or asks about evaluating this task. Reports Task Success Rate (TSR).

- Skill: `qhjqhj00/mobibench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/mobibench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mobibench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/mobibench-eval

---


# mobibench-eval

> Modular and Multi-Path-Aware Offline Benchmarking for Mobile GUI Agents — Im et al. (2025) (arXiv:2512.12634, 2025)

## What this evaluates

Evaluates mobile GUI agents' ability to complete tasks on mobile interfaces by measuring task success rate across diverse, multi-path trajectories. It also enables modular component analysis (screen parsing, history generation, inference style, reflection) to identify performance bottlenecks and optimal configurations for different foundation models.

## Datasets

- **MobiBench** — total 105; splits: test (105); repo https://github.com/fclab-skku/Mobi-Bench

## Metrics

- `Task Success Rate (TSR)` **(primary)** — range: percent
  - Percentage of tasks completed successfully. Calculated as (number of successfully completed tasks / total number of tasks) * 100. Ground truth is validated by human judges with 93.01% inter-annotator agreement.
- `Action Accuracy (A.Acc)` — range: [0, 1]
  - Percentage of individual steps/actions predicted correctly by the agent at each turn.
- `Cost` — range: other
  - Monetary cost per task in USD, calculated based on the input/output token pricing of the underlying foundation model.

## Input / output format

**Input**: Task instruction (natural language goal) + current mobile UI state (screenshot and/or accessibility tree) + conversation history.

**Output**: Sequence of GUI actions (e.g., click, type, scroll, back) or a completion signal.

## Scoring recipe

```python
def compute_tsr(predictions, valid_paths):
    correct = 0
    for pred_trace, paths in zip(predictions, valid_paths):
        if any(trace_matches(pred_trace, path) for path in paths):
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Single-path offline benchmarks systematically penalize agents for choosing equally valid alternative paths, leading to artificially low success rates.
- Naive self-reflection often backfires due to false alarms, incorrectly modifying correct actions and degrading overall performance.
- Assuming ReAct prompting or Set-of-Mark are universally optimal can lead to suboptimal configurations; effectiveness heavily depends on the underlying model size and type.

## Evidence (verbatim from paper)

> Table 4 reports the Task Success Rate (TSR) of m3a under four different evaluation settings. ... For ground truth, three human judges manually validated the agent’s full execution traces (inter-annotator agreement: 93.01%).

## Citation

```bibtex
@misc{im2025mobibench,
  title={Modular and Multi-Path-Aware Offline Benchmarking for Mobile GUI Agents},
  author={Im et al. (2025)},
  year={2025},
  note={arXiv:2512.12634}
}
```

- arXiv: 2512.12634