# Mobileagentbench Eval

> This benchmark evaluates the performance of LLM-based mobile agents on Android GUI navigation tasks. It measures end-to-end task completion, action efficiency, response latency, computational cost, and the agent's ability to correctly determine task completion without stopping too early or too late. Use when the user wants to benchmark on MobileAgentBench, or asks about evaluating this task. Reports Success Rate (SR).

- Skill: `qhjqhj00/mobileagentbench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/mobileagentbench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mobileagentbench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/mobileagentbench-eval

---


# mobileagentbench-eval

> MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents — Wang et al. (2024) (arXiv:2406.08184, 2024)

## What this evaluates

This benchmark evaluates the performance of LLM-based mobile agents on Android GUI navigation tasks. It measures end-to-end task completion, action efficiency, response latency, computational cost, and the agent's ability to correctly determine task completion without stopping too early or too late.

## Datasets

- **MobileAgentBench** — total 100; splits: test (100)

## Metrics

- `Success Rate (SR)` **(primary)** — range: [0, 1]
  - SR = N_success / M_tasks, where N_success is the number of tasks successfully completed and M_tasks is the total number of benchmarking tasks. Reflects end-to-end task completion ability.
- `Step-wise Efficiency (SE)` — range: [0, 1]
  - SE = S_actual / S_min, where S_actual is the number of steps taken to successfully finish a task and S_min is the minimum required steps. Only calculated for successful tasks.
- `Latency` — range: other
  - Average time in seconds spent before and after each action. Measures user wait time between consecutive steps.
- `Tokens` — range: other
  - Total number of LLM input and output tokens. Calculated using a GPT-4V approximation standard (4 chars/token for text; 170 tokens + 85 base per 512x512 image tile).
- `False Negative (FN) Rate` — range: [0, 1]
  - FN = N_early / M_failure, where N_early is the number of tasks stopped prematurely and M_failure is the total number of failed tasks. Measures likelihood of falsely thinking a task is finished.
- `False Positive (FP) Rate` — range: [0, 1]
  - FP = N_late / M_success, where N_late is the number of tasks stopped after completion and M_success is the total number of successful tasks. Measures likelihood of falsely thinking a task is not finished.

## Input / output format

**Input**: Current mobile UI screenshot, task instruction, and optionally the textual view hierarchy of the screen.

**Output**: A discrete action command (e.g., click, type, scroll, back, or done) to execute next on the interface.

## Scoring recipe

```python
def compute_metrics(predictions, golds, actual_steps, action_times, token_counts):
    M_tasks = len(golds)
    N_success = sum(1 for p, g in zip(predictions, golds) if p == g)
    SR = N_success / M_tasks

    successful_steps = [s for s, p in zip(actual_steps, predictions) if p == golds[predictions.index(p)]]
    SE = sum(successful_steps) / len(successful_steps) if successful_steps else 0

    Latency = sum(action_times) / len(action_times)
    Tokens = sum(token_counts) / len(token_counts)

    M_failure = M_tasks - N_success
    N_early = sum(1 for p in predictions if p == 'early_stop' and p != golds[predictions.index(p)])
    FN = N_early / M_failure if M_failure > 0 else 0

    N_late = sum(1 for p in predictions if p == 'late_stop' and p == golds[predictions.index(p)])
    FP = N_late / N_success if N_success > 0 else 0
    return {'SR': SR, 'SE': SE, 'Latency': Latency, 'Tokens': Tokens, 'FN Rate': FN, 'FP Rate': FP}
```

## Common pitfalls

- Step-wise Efficiency (SE) is calculated only on successfully completed tasks, ignoring the inefficiency of failed attempts.
- The benchmark caps execution steps at twice the minimum required, which artificially restricts error correction on easy tasks while allowing more flexibility on harder ones.
- Token counts use a fixed GPT-4V approximation rule rather than the actual model's tokenizer, potentially misrepresenting real inference costs.
- Early/late stopping rates (FN/FP) are computed relative to failure/success counts respectively, meaning a high FP rate can occur even if overall success is high.

## Evidence (verbatim from paper)

> We define 6 metrics to comprehensively benchmarking mobile agents: Success Rate (SR): ${SR}\=N_{success}/M_{tasks}$, where $N_{success}$ is the number of successful tasks, judged by the benchmark system. $M_{tasks}$ is the number of total benchmarking tasks. This metric reflects the agent’s ability to correctly finish a task end-to-end.

## Citation

```bibtex
@misc{wang2024mobileagentbench,
  title={MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2406.08184}
}
```

- arXiv: 2406.08184