Mobile Agent V2 Eval

mobile-agent-v2-eval

Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration — Wang et al. (2024) (arXiv:2406.01014, 2024)

What this evaluates

Probes an agent's ability to navigate and execute multi-step UI operations on real mobile devices (Android/HarmonyOS) based on natural language instructions. It evaluates end-to-end task completion, step-level correctness, decision-making precision, and the capacity to detect and correct operational errors via reflection.

Datasets

• Mobile-Agent-v2 Evaluation Set — total 88; splits: test (88)

Metrics

• Success Rate (SR) (primary) — range: percent
  • Proportion of user instructions where all requirements are fully fulfilled by the agent's execution. Calculated as 1.0 if the predicted action sequence exactly matches the ground truth, else 0.0, averaged over all instructions.
• Completion Rate (CR) — range: percent
  • Proportion of correct steps out of the total ground truth operations. Calculated as the count of matching steps divided by the length of the ground truth sequence.
• Decision Accuracy (DA) — range: percent
  • Proportion of correct decisions made by the decision agent out of all decisions issued during task execution.
• Reflection Accuracy (RA) — range: percent
  • Proportion of correct reflections made by the reflection agent out of all reflection attempts, measuring error detection and correction capability.

Input / output format

Input: Natural language user instruction, current device screen screenshot, and optional operation history/memory context.

Output: Sequence of UI actions (e.g., click, type, scroll) or reflection/decision tokens, executed via ADB on the target OS.

Scoring recipe

def compute_metrics(predicted_actions, ground_truth_actions, predicted_decisions, predicted_reflections):
    # Success Rate: exact match of full action sequence
    sr = 1.0 if predicted_actions == ground_truth_actions else 0.0
    # Completion Rate: step-wise correctness
    cr = sum(p == g for p, g in zip(predicted_actions, ground_truth_actions)) / len(ground_truth_actions)
    # Decision Accuracy: correct decisions / total decisions
    da = sum(1 for d in predicted_decisions if d.is_correct) / len(predicted_decisions)
    # Reflection Accuracy: correct reflections / total reflections
    ra = sum(1 for r in predicted_reflections if r.is_correct) / len(predicted_reflections)
    return {'SR': sr, 'CR': cr, 'DA': da, 'RA': ra}

Common pitfalls

• Evaluation requires real devices or ADB emulators, making it resource-intensive and harder to reproduce than screenshot-only benchmarks.
• The paper explicitly rejects the standard Mobile-Eval benchmark due to low difficulty, so results are not directly comparable to prior mobile agent papers using that dataset.
• Knowledge injection is treated as an ablation/variant rather than a standard setting, which can confuse baseline comparisons if not carefully separated.

Evidence (verbatim from paper)

Metrics. We design the following four metrics for dynamic evaluation: Success Rate (SR): When all the requirements of a user instruction are fulfilled, the agent is considered to have successfully executed this instruction. The success rate refers to the proportion of user instructions that are successfully executed. Completion Rate (CR): Although some challenging instructions may not be successfully executed, the correct operations performed by the agent are still noteworthy. The completion rate refers to the proportion of correct steps out of the ground truth operations. Decision Accuracy (DA): This metric reflects the accuracy of the decision by the decision agent. It is the proportion of correct decisions out of all decisions. Reflection Accuracy (RA): This metric reflects the accuracy of reflection by the reflection agent. It is the proportion of correct reflections out of all reflections.

Citation

@misc{wang2024mobileagentv2,
  title={Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2406.01014}
}

• arXiv: 2406.01014