Spatial457 Eval

spatial457-eval

Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Models — Wang et al. (2025) (arXiv:2502.08636, 2025)

What this evaluates

This benchmark evaluates large multimodal models' ability to perform 6D spatial reasoning across multiple difficulty levels. It probes capabilities including multi-object recognition, 2D and 3D location understanding, 3D orientation interpretation, and occlusion/collision prediction. It also quantifies systematic prediction biases across visual attributes like color, shape, size, and pose.

Datasets

• Spatial457 — total 1000; splits: test (1000); repo https://github.com/XingruiWang/Spatial457

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions out of the total number of questions per category or difficulty level.
• RPDR — range: percent
  • Relative Performance Dropping Rate; quantifies the decline in accuracy when introducing additional spatial factors or complexity levels.
• CV — range: other
  • Coefficient of Variation for prediction bias; calculated as the standard deviation divided by the mean of the predicted probability distribution for each attribute label.

Input / output format

Input: A synthetic 3D-rendered image paired with a natural language question probing spatial relationships, object attributes, or pose.

Output: A text response containing the predicted answer (and optionally reasoning steps) to the posed question.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for p, g in zip(predictions, gold_answers):
        if normalize_text(p) == normalize_text(g):
            correct += 1
    return (correct / len(gold_answers)) * 100

def compute_cv(attribute_predictions):
    # attribute_predictions: dict mapping label -> count/prob
    probs = list(attribute_predictions.values())
    mean_p = sum(probs) / len(probs)
    std_p = (sum((x - mean_p)**2 for x in probs) / len(probs)) ** 0.5
    return std_p / mean_p if mean_p > 0 else 0

Common pitfalls

• Models exhibit strong attribute biases (e.g., favoring 'yellow' for color or 'front' for pose), which can artificially inflate accuracy if the dataset were not synthetically balanced.
• Performance drops sharply from 2D to 3D/6D tasks; evaluating only on 2D spatial relationships will significantly overestimate a model's true spatial reasoning capability.
• The benchmark uses synthetic 3D renders; results may not directly transfer to real-world photographic inputs without domain adaptation.

Evidence (verbatim from paper)

We generate 1,000 images to test the performance of large vision-language models. For the seven types of questions, we generate 670 single-object questions (L1-Single), 4,995 multiple-object questions (L2-Multi-obj.), 4,995 2D spatial relationship questions (L3-2D-Spatial), 3,534 occlusion questions (L4-Occlusion), 4,555 3D pose questions (L4-3D-Pose), 3,166 collision questions (L5-Collision), and 1,837 6D spatial relationship questions (L5-6D-Spatial)... The RPDR analysis provides a structured approach to quantify the decline in accuracy with the introduction of each new factor, as detailed in [Tab. 3].

Citation

@misc{wang2025spatial457,
  title={Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Models},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2502.08636}
}

• arXiv: 2502.08636