Visuriddles Eval

visuriddles-eval

VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning — Yan et al. (2025) (arXiv:2506.02537, 2025)

What this evaluates

Evaluates multimodal large language models' ability to perform abstract visual reasoning across five fine-grained perceptual dimensions (numerosity, attributes, style, position, spatial relations) and two high-level reasoning tasks (analogical pattern matching and constraint-based logic).

Datasets

• VisuRiddles — total 1000; splits: test (1000); repo https://github.com/yh-hust/VisuRiddles

Metrics

• exact match (primary) — range: [0, 1]
  • Fraction of correctly predicted answers. For single-choice tasks, correct if the predicted option matches the gold option. For high-level tasks, correct if the generated symbolic output exactly matches the gold output.

Input / output format

Input: Image grid(s) (e.g., 3x3 matrix or progression) accompanied by a text question in English.

Output: For basic categories: a single letter (A, B, C, or D). For high-level categories: exact symbolic output (constraint-based grid or pattern match index).

Scoring recipe

def compute_metric(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if len(gold) > 1:  # high-level symbolic
            correct += (pred.strip() == gold.strip())
        else:  # single-choice
            correct += (pred.strip().upper() == gold.strip().upper())
    return correct / len(golds)

Common pitfalls

• Assuming all tasks use multiple-choice format; high-level RAVEN and Sudoku tasks require exact symbolic generation, not letter selection.
• Overestimating real-world difficulty; the dataset is heavily synthesized via the PRS framework with deliberately tractable complexity to focus on perceptual supervision.

Evidence (verbatim from paper)

In the high-level categories, RAVEN Reasoning and Sudoku Reasoning each account for 10% of the total data, and require models to generate exact symbolic outputs to be considered correct.

Citation

@misc{yan2025visuriddles,
  title={VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning},
  author={Yan et al. (2025)},
  year={2025},
  note={arXiv:2506.02537}
}

• arXiv: 2506.02537