Mas Bench Eval

mas-bench-eval

MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agents — Zhao et al. (2025) (arXiv:2509.06477, 2025)

What this evaluates

Evaluates the ability of mobile GUI agents to complete complex, real-world automation tasks across single-app and cross-app scenarios. It specifically probes how well agents can integrate predefined or self-generated shortcuts (APIs, deep links, RPA scripts) with standard GUI interactions to improve task success, execution efficiency, and cost-effectiveness.

Datasets

• MAS-Bench — total 139; splits: test (139)

Metrics

• SR (primary) — range: percent
  • Success Rate: proportion of tasks completed successfully out of the total 139 tasks.
• MSRS — range: [0, 1]
  • Mean Step Ratio on Successful tasks: average number of steps taken by the agent on successfully completed tasks, normalized against an optimal baseline path.
• MET — range: seconds
  • Mean Execution Time: average time in seconds required to complete a task.
• MToC — range: kTokens
  • Mean Token Cost: average number of tokens consumed per task, reported in thousands (kTokens).
• SSR — range: percent
  • Shortcut Success Rate: proportion of shortcut calls (APIs, deep links, RPA scripts) that execute successfully without error.

Input / output format

Input: Screenshot (SS) and/or View Hierarchy (UI Tree) of the mobile device screen, along with the task instruction. Base model used is Gemini-2.5-Pro.

Output: Sequential GUI actions (e.g., click, type, swipe, back) or shortcut invocations (API calls, deep links, RPA scripts) mapped to a standardized action space.

Scoring recipe

def compute_metrics(predictions, golds):
    total = len(golds)
    successful = [p for p, g in zip(predictions, golds) if p['success']]
    sr = len(successful) / total
    msrs = sum(p['steps'] for p in successful) / len(successful) if successful else 0
    met = sum(p['time_sec'] for p in successful) / len(successful) if successful else 0
    mtoC = sum(p['tokens_k'] for p in successful) / len(successful) if successful else 0
    shortcut_calls = [p for p in predictions if p.get('shortcut_called')]
    ssr = sum(1 for p in shortcut_calls if p['shortcut_success']) / len(shortcut_calls) if shortcut_calls else 0
    return {'SR': sr, 'MSRS': msrs, 'MET': met, 'MToC': mtoC, 'SSR': ssr}

Common pitfalls

• MSRS is calculated only on successful tasks, which can artificially inflate efficiency scores if agents frequently terminate early on failures.
• Shortcut Success Rate (SSR) and Task Success Rate (SR) are distinct; a failed shortcut does not necessarily mean task failure if the agent falls back to GUI actions.
• Token cost (MToC) and execution time (MET) are highly sensitive to the base LLM's context window and API latency, making cross-model comparisons require identical base models.

Evidence (verbatim from paper)

Table 2: Performance comparison of our MAS agents and baseline methods on MAS-Bench with a predefined shortcuts knowledge base. All agents utilize the Gemini-2.5-Pro. SS and VH refer to the Screenshot and View Hierarchy (UI Tree) input modalities. MSRS is the Mean Step Ratio on Successful tasks, MET is the Mean Execution Time in seconds, and MToC is the Mean Token Cost in thousands (kTokens).

Citation

@misc{zhao2025masbench,
  title={MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agents},
  author={Zhao et al. (2025)},
  year={2025},
  note={arXiv:2509.06477}
}

• arXiv: 2509.06477