Mobileworld Eval

mobileworld-eval

MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments — Kong et al. (2025) (arXiv:2512.19432, 2025)

What this evaluates

Evaluates autonomous mobile agents on long-horizon, cross-application workflows, requiring them to handle ambiguous instructions via agent-user interaction and integrate external tools via MCP. It probes planning, GUI grounding, clarification strategies, and tool orchestration in real-world mobile environments.

Datasets

• MobileWorld — total 201; splits: test (201)

Metrics

• Success Rate (SR) (primary) — range: percent
  • SR = (1/N) * sum(s_i) where s_i is 1 if the task objective is fully achieved, else 0. Computed over all tasks or specific categories (GUI-Only, Interaction, MCP).
• Average Completion Steps (Ave. Steps) — range: other
  • Ave. Steps = (1/N) * sum(t_i) where t_i is the number of action steps in the trajectory for task i, including failed episodes.
• Average User Queries (Ave. Queries) — range: other
  • Ave. Queries = (1/|I_interact|) * sum(c_i) where c_i is the number of ask_user actions invoked for agent-user interaction tasks.
• User Interaction Quality (UIQ) — range: [0, 1]
  • UIQ = sum(q_i) / (|I_interact| + |I_triggered|) where q_i = s_i/c_i if c_i>0 else 0. Rewards successful interaction tasks with fewer queries and penalizes unnecessary ask_user on non-interaction tasks.
• Average MCP Tool Calls (Ave. MCP Calls) — range: other
  • Ave. MCP Calls = (1/|I_MCP|) * sum(m_i) where m_i is the number of MCP tool calls for MCP-augmented tasks.

Input / output format

Input: Task instruction, current application screenshot, interaction history (including previous actions, user responses, or MCP tool outputs), and MCP tool specifications (for MCP-augmented tasks).

Output: Planner outputs an action type (click, swipe, wait, type, ask_user, mcp_call) and a natural language description of the target UI element or MCP tool parameters. Executor outputs precise pixel coordinates for click/long_press actions.

Scoring recipe

def score(predictions, max_steps=50):
    sr = sum(1 for p in predictions if p['success']) / len(predictions)
    steps = sum(p['steps'] for p in predictions) / len(predictions)
    interact = [p for p in predictions if p['category'] == 'interact']
    queries = sum(p['ask_user_count'] for p in interact) / len(interact) if interact else 0
    triggered = [p for p in predictions if p['category'] != 'interact' and p['ask_user_count'] > 0]
    uiq_vals = [(1 if p['success'] else 0) / p['ask_user_count'] if p['ask_user_count'] > 0 else 0 for p in interact + triggered]
    uiq = sum(uiq_vals) / (len(interact) + len(triggered)) if (interact or triggered) else 0
    mcp = [p for p in predictions if p['category'] == 'mcp']
    mcp_calls = sum(p['mcp_call_count'] for p in mcp) / len(mcp) if mcp else 0
    return {'SR': sr, 'Ave_Steps': steps, 'Ave_Queries': queries, 'UIQ': uiq, 'Ave_MCP_Calls': mcp_calls}

Common pitfalls

• Models often terminate early upon failure, which artificially lowers Average Completion Steps but also reduces Success Rate, making efficiency metrics misleading without context.
• UIQ penalizes agents for invoking ask_user on non-interaction tasks, so agents may avoid necessary clarifications to maintain a high score.
• MCP tool invocation failures due to incorrect names/arguments still count towards Ave. MCP Calls but do not contribute to MCP Success Rate, obscuring tool usage effectiveness.

Evidence (verbatim from paper)

We define the following metrics to measure the effectiveness and efficiency of GUI agents on MobileWorld. Success Rate (SR) measures the proportion of tasks successfully completed by the agent. For each task $i$ in the MobileWorld, a binary score $s_{i}\in{0,1}$ is assigned, where $s_{i}=1$ if the task objective is fully achieved and $s_{i}=0$ otherwise. The success rate is computed as: SR = (1/N) \sum_{i=1}^{N} s_{i}.

Citation

@misc{kong2025mobileworld,
  title={MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments},
  author={Kong et al. (2025)},
  year={2025},
  note={arXiv:2512.19432}
}

• arXiv: 2512.19432