qhjqhj00 m-portal-eval
MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning — Jiang et al. (2025) (arXiv:2506.22992, 2025)
What this evaluates
Evaluates multi-step spatial and physical reasoning in multimodal language models by requiring them to generate or validate chain-of-thought plans for solving Portal 2-inspired puzzle maps. The benchmark probes the model's ability to integrate visual map layouts with textual instructions to produce physically sound, multi-step traversal strategies.
Datasets
- M-Portal — total 1024; splits: test (1024)
Metrics
F1 score(primary) — range: percent- Binary classification F1-score for the plan correctness subtask, measuring the harmonic mean of precision and recall for identifying correct vs. incorrect candidate plans.
Accuracy— range: percent- Exact match accuracy for the fill-the-blanks subtask, measuring the proportion of instances where the model correctly selects and orders the missing steps from a set of candidate options.
Input / output format
Input: Multimodal inputs (screenshots of a Portal map and textual contextualization) plus a task instruction. For fill-the-blanks: a partial plan with masked steps and a set of candidate options.
Output: A Chain-of-Thought plan consisting of interpretable reasoning steps (s1, s2, ..., sn) for plan correctness; or a selection of correct missing steps in order for fill-the-blanks.
Scoring recipe
# Plan Correctness (512 instances)
correct = 0
for plan in model_plans:
if plan == ground_truth_plan:
correct += 1
precision = correct / len(model_plans)
recall = correct / len(ground_truth_plans)
f1 = 2 * precision * recall / (precision + recall)
# Fill-the-blanks (512 instances)
correct = 0
for selection in model_selections:
if selection == ground_truth_steps:
correct += 1
accuracy = correct / len(model_selections)
Common pitfalls
- High class imbalance in plan correctness: each map yields 1 correct plan and 32 incorrect ones, skewing majority-class accuracy.
- Evaluation is closed-ended (binary/multiple-choice) rather than interactive; models are not tested in the actual game environment.
- Models frequently fail at the perception stage, misinterpreting visual map layouts, which cascades into reasoning failures even on simplified subtasks.
Evidence (verbatim from paper)
On the plan correctness task, all investigated models (MLLMs as well as text-only LLMs) performed very poorly with a minority class F1 score of around 6%, similar to the random baseline. In the easier fill-the-blanks task, 8 out of 12 models outperform the random baseline.
Citation
@misc{jiang2025marble,
title={MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning},
author={Jiang et al. (2025)},
year={2025},
note={arXiv:2506.22992}
}
- arXiv: 2506.22992