Mobileagentbench Eval

mobileagentbench-eval

MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents — Wang et al. (2024) (arXiv:2406.08184, 2024)

What this evaluates

This benchmark evaluates the performance of LLM-based mobile agents on Android GUI navigation tasks. It measures end-to-end task completion, action efficiency, response latency, computational cost, and the agent's ability to correctly determine task completion without stopping too early or too late.

Datasets

• MobileAgentBench — total 100; splits: test (100)

Metrics

• Success Rate (SR) (primary) — range: [0, 1]
  • SR = N_success / M_tasks, where N_success is the number of tasks successfully completed and M_tasks is the total number of benchmarking tasks. Reflects end-to-end task completion ability.
• Step-wise Efficiency (SE) — range: [0, 1]
  • SE = S_actual / S_min, where S_actual is the number of steps taken to successfully finish a task and S_min is the minimum required steps. Only calculated for successful tasks.
• Latency — range: other
  • Average time in seconds spent before and after each action. Measures user wait time between consecutive steps.
• Tokens — range: other
  • Total number of LLM input and output tokens. Calculated using a GPT-4V approximation standard (4 chars/token for text; 170 tokens + 85 base per 512x512 image tile).
• False Negative (FN) Rate — range: [0, 1]
  • FN = N_early / M_failure, where N_early is the number of tasks stopped prematurely and M_failure is the total number of failed tasks. Measures likelihood of falsely thinking a task is finished.
• False Positive (FP) Rate — range: [0, 1]
  • FP = N_late / M_success, where N_late is the number of tasks stopped after completion and M_success is the total number of successful tasks. Measures likelihood of falsely thinking a task is not finished.

Input / output format

Input: Current mobile UI screenshot, task instruction, and optionally the textual view hierarchy of the screen.

Output: A discrete action command (e.g., click, type, scroll, back, or done) to execute next on the interface.

Scoring recipe

def compute_metrics(predictions, golds, actual_steps, action_times, token_counts):
    M_tasks = len(golds)
    N_success = sum(1 for p, g in zip(predictions, golds) if p == g)
    SR = N_success / M_tasks

    successful_steps = [s for s, p in zip(actual_steps, predictions) if p == golds[predictions.index(p)]]
    SE = sum(successful_steps) / len(successful_steps) if successful_steps else 0

    Latency = sum(action_times) / len(action_times)
    Tokens = sum(token_counts) / len(token_counts)

    M_failure = M_tasks - N_success
    N_early = sum(1 for p in predictions if p == 'early_stop' and p != golds[predictions.index(p)])
    FN = N_early / M_failure if M_failure > 0 else 0

    N_late = sum(1 for p in predictions if p == 'late_stop' and p == golds[predictions.index(p)])
    FP = N_late / N_success if N_success > 0 else 0
    return {'SR': SR, 'SE': SE, 'Latency': Latency, 'Tokens': Tokens, 'FN Rate': FN, 'FP Rate': FP}

Common pitfalls

• Step-wise Efficiency (SE) is calculated only on successfully completed tasks, ignoring the inefficiency of failed attempts.
• The benchmark caps execution steps at twice the minimum required, which artificially restricts error correction on easy tasks while allowing more flexibility on harder ones.
• Token counts use a fixed GPT-4V approximation rule rather than the actual model's tokenizer, potentially misrepresenting real inference costs.
• Early/late stopping rates (FN/FP) are computed relative to failure/success counts respectively, meaning a high FP rate can occur even if overall success is high.

Evidence (verbatim from paper)

We define 6 metrics to comprehensively benchmarking mobile agents: Success Rate (SR): ${SR}=N_{success}/M_{tasks}$, where $N_{success}$ is the number of successful tasks, judged by the benchmark system. $M_{tasks}$ is the number of total benchmarking tasks. This metric reflects the agent’s ability to correctly finish a task end-to-end.

Citation

@misc{wang2024mobileagentbench,
  title={MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2406.08184}
}

• arXiv: 2406.08184