Mobile Eval

mobile-eval

Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception — Wang et al. (2024) (arXiv:2401.16158, 2024)

What this evaluates

Evaluates an autonomous mobile device agent's ability to execute multi-step UI operations using visual perception. It probes task planning, self-reflection, and cross-application interaction under varying instruction complexity.

Datasets

• Mobile-Eval — total 30; splits: test (30)

Metrics

• Success (Su) (primary) — range: [0, 1]
  • Boolean; 1 if the agent successfully completes the user instruction, 0 otherwise.
• Process Score (PS) — range: [0, 1]
  • Number of correct execution steps divided by the total number of steps taken by the agent.
• Relative Efficiency (RE) — range: other
  • Reported as a ratio of agent steps to human baseline steps (agent_steps / human_steps). Human steps are manually recorded as the optimal solution.
• Completion Rate (CR) — range: [0, 1]
  • Number of human-optimal steps successfully executed by the agent divided by the total number of human-optimal steps.

Input / output format

Input: Natural language instruction paired with real-time mobile screen screenshots (visual perception).

Output: Sequence of discrete UI operations (e.g., click, type, swipe, navigate) executed on the device.

Scoring recipe

def compute_metrics(agent_actions, human_actions, success_flag):
    su = 1.0 if success_flag else 0.0
    ps = count_correct_steps(agent_actions) / len(agent_actions) if len(agent_actions) > 0 else 0.0
    re = f"{len(agent_actions)} / {len(human_actions)}"
    cr = count_human_steps_completed(agent_actions) / len(human_actions) if len(human_actions) > 0 else 0.0
    return su, ps, re, cr

Common pitfalls

• Process Score (PS) can remain high even when the final task fails, as it only measures intermediate step accuracy rather than end-to-end success.
• Relative Efficiency (RE) is reported as a raw step ratio rather than a normalized percentage, which can exceed 1.0 if the agent takes more steps than the human baseline.
• Completion Rate (CR) is capped at 1.0 and does not penalize the agent for taking extra or redundant steps beyond the human baseline.

Evidence (verbatim from paper)

Metrics. We have designed four metrics to assess the performance of the Mobile-Agent from different perspectives:

• •

  Success (Su): If the Mobile-Agent completes the instruction, it is considered successful.

• •

  Process Score (PS): This metric measures the accuracy of each step in the execution of instructions. Specifically, it equals the number of correct steps divided by the total number of steps.

Citation

@misc{wang2024mobileagent,
  title={Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2401.16158}
}

• arXiv: 2401.16158