Robobench Eval

robobench-eval

Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain — Luo et al. (2025) (arXiv:2510.17801, 2025)

What this evaluates

Evaluates multimodal large language models (MLLMs) as embodied brains in robotic manipulation. It probes five cognitive dimensions: instruction comprehension, perception reasoning, generalized planning, affordance prediction, and failure analysis across diverse real-world robotic tasks.

Datasets

• RoboBench — total 6092; splits: test (6092)

Metrics

• accuracy (%) (primary) — range: percent
  • Calculated as the percentage of correctly answered questions out of the total number of questions per dimension. Scores are reported for each of the five cognitive dimensions and averaged for an overall score.

Input / output format

Input: Multimodal inputs consisting of real-world robotic scene images or videos paired with natural language instructions or questions.

Output: Text-based answers to the provided questions, evaluated for correctness against ground-truth labels.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

# Report per dimension and average across the five dimensions

Common pitfalls

• Models show a ~30% performance drop on implicit instructions compared to explicit ones, indicating weak intent grounding.
• Spatiotemporal reasoning and causal grounding across time are consistently weak, with temporal grounding accuracy often below 50%.
• Cross-embodiment planning (e.g., dual-arm coordination, mobile manipulation) remains a major bottleneck for current MLLMs.

Evidence (verbatim from paper)

Perception Challenges: While models exhibit reasonable performance in object property analysis, they struggle with fundamental robotic perception and spatiotemporal reasoning. Common failures include misidentifying the robot type or viewpoint, and being unable to accurately localize events in time. In Table 3, the success rate on robotic-view recognition is only 43.48, and temporal grounding accuracy reaches merely 49.68.

Citation

@misc{luo2025robobench,
  title={Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain},
  author={Luo et al. (2025)},
  year={2025},
  note={arXiv:2510.17801}
}

• arXiv: 2510.17801