Magma Agentic Eval

magma-agentic-eval

Magma: A Foundation Model for Multimodal AI Agents — Jianwei Yang et al. (arXiv:2502.13130, 2025)

What this evaluates

This protocol evaluates a model's capability to perform multimodal agentic tasks, specifically UI navigation and robotic manipulation. It probes spatial-temporal reasoning, action grounding, and zero-shot or few-shot transfer across digital interfaces and physical simulators.

Datasets

• ScreenSpot — total ?; splits: test (-1)
• VisualWebBench — total ?; splits: test (-1)
• SimplerEnv — total ?; splits: test (-1)
• Mind2Web — total ?; splits: test (-1)
• AITW — total ?; splits: test (-1)
• LIBERO — total ?; splits: test (-1)

Metrics

• element_selection_accuracy — range: [0, 1]
  • Percentage of correctly predicted UI elements out of total ground-truth elements.
• operation_f1 — range: [0, 1]
  • Token-wise F1 score between the predicted operation sequence and the ground-truth operation sequence.
• step_success_rate (primary) — range: [0, 1]
  • Percentage of tasks where the model successfully completes all required steps in sequence.
• success_rate — range: [0, 1]
  • Percentage of successful trials out of total trials (typically 10 per task) for robotic manipulation.

Input / output format

Input: UI screenshots (optionally annotated with Set-of-Mark labels) or robot visual observations and state.

Output: Predicted UI element coordinates and operation text, or robot action trajectories (e.g., 7-DoF commands).

Scoring recipe

def score_ui(predictions, gold):
    ele_acc = sum(1 for p, g in zip(predictions.elements, gold.elements) if p == g) / len(gold)
    op_f1 = token_f1_score(predictions.operations, gold.operations)
    step_sr = sum(1 for p, g in zip(predictions.steps, gold.steps) if p == g) / len(gold)
    return {'element_selection_accuracy': ele_acc, 'operation_f1': op_f1, 'step_success_rate': step_sr}

def score_robotics(predictions, gold):
    success = sum(1 for trial in predictions.trials if trial.reaches_goal(gold.goal_state))
    return success / len(predictions.trials)

Common pitfalls

• Confusing zero-shot pretrained performance with finetuned performance; the paper reports both separately and notes significant gaps.
• Assuming DOM tree input is required for UI tasks; Magma uses image-only input augmented with Set-of-Mark prompting.
• Mixing up UI navigation metrics (Ele. Acc, Op. F1, Step SR) with robotic success rates, as they measure different granularities of task completion.

Evidence (verbatim from paper)

“Ele. Acc” denotes element selection accuracy. “Op. F1” denotes the token-wise F1 score between predicted ground-truth operation. “Step SR” denotes the step-wise success rate.

Citation

@misc{yang2025magma,
  title={Magma: A Foundation Model for Multimodal AI Agents},
  author={Jianwei Yang et al.},
  year={2025},
  note={arXiv:2502.13130}
}

• arXiv: 2502.13130