M Cube Eval

m-cube-eval

MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning — Jiang et al. (2025) (arXiv:2506.22992, 2025)

What this evaluates

Evaluates 3D spatial reasoning and combinatorial planning by requiring models to assemble jigsaw-style pieces into a 5x5x5 cube under physical constraints. The benchmark probes the model's ability to extract geometric patterns from rendered views and logically arrange pieces without gaps or overlaps.

Datasets

• M-Cube — total 2000; splits: test (2000)

Metrics

• binary_evaluation (primary) — range: percent
  • Binary success rate (0 or 1) indicating whether the model's proposed piece-to-face mapping and orientations successfully assemble into a perfect cube without gaps or overlaps, validated against geometric constraints and rotational symmetries.

Input / output format

Input: Rendered 3D images of 6 jigsaw-style pieces from a random viewpoint (elevation -155° to -115°, azimuth -150° to -90°), or 2D arrays for the CUBE-easy variant. CUBE-easy also provides a partial solution with 4 pieces already placed.

Output: A piece-to-face assignment and orientation (rotation/flip) for each of the 6 pieces.

Scoring recipe

# For each of the 2000 instances
success_count = 0
for config in model_configs:
    if solution_validator(config):  # Checks edge alignment, gap/overlap, and rotational equivalence
        success_count += 1
accuracy = success_count / 2000

Common pitfalls

• No unique ground-truth answer due to 24 rotational symmetries; exact string matching fails, requiring a geometric validator.
• Perception bottleneck: models struggle to extract bump/gap patterns from rendered images, causing immediate failure before reasoning begins.
• Combinatorial explosion: 6! piece assignments × 8 orientations per piece makes random guessing statistically negligible.

Evidence (verbatim from paper)

Therefore, instead of directly comparing the answer to ground-truth, we provide a solution validator by testing whether the solution from MLLM could successfully assemble the pieces into a perfect cube. Beside binary evaluation, the solution validator could also identify the conflicts in a given configuration, such as mismatched edges.

Citation

@misc{jiang2025marble,
  title={MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning},
  author={Jiang et al. (2025)},
  year={2025},
  note={arXiv:2506.22992}
}

• arXiv: 2506.22992