qhjqhj00 dori-eval
Right Side Up? Disentangling Orientation Understanding in MLLMs with Fine-grained Multi-axis Perception Tasks — Nichols et al. (2025) (arXiv:2505.21649, 2025)
What this evaluates
Evaluates multimodal large language models' ability to understand and reason about object orientation across four dimensions: frontal alignment, rotational transformations, relative directional relationships, and canonical orientation. It distinguishes between coarse categorical judgments and fine-grained angular estimations to probe 3D spatial reasoning capabilities.
Datasets
- DORI — total 33656; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correct answers chosen from multiple options. Calculated as (number of correct predictions / total number of questions) × 100.
Input / output format
Input: Image input paired with a standardized multiple-choice question prompt describing an orientation task (e.g., frontal alignment, rotation, relative direction, or canonical orientation).
Output: Model must output a selected answer option from the provided multiple-choice set, along with a reasoning explanation.
Scoring recipe
correct = 0
total = len(predictions)
for pred, gold in zip(predictions, gold_labels):
if pred.strip().lower() == gold.strip().lower():
correct += 1
accuracy = (correct / total) * 100
Common pitfalls
- Models often perform worse than random chance on certain sub-tasks, indicating systematic failure rather than mere noise.
- Performance heavily depends on whether the question requires coarse categorical judgment or fine-grained angular estimation, with a large drop in the latter.
- Results may be inflated for datasets like COCO due to potential train/test distribution overlap from pretraining data.
Evidence (verbatim from paper)
We evaluate models on accuracy—the percentage of correct answers chosen from multiple options—across both coarse and granular question types. All models were prompted with the standardized task descriptions and prompted to provide both an answer selection and a reasoning explanation.
Citation
@misc{nichols2025dori,
title={Right Side Up? Disentangling Orientation Understanding in MLLMs with Fine-grained Multi-axis Perception Tasks},
author={Nichols et al. (2025)},
year={2025},
note={arXiv:2505.21649}
}
- arXiv: 2505.21649