Mm Iq Eval

mm-iq-eval

MM-IQ: Benchmarking Human-Like Abstraction and Reasoning in Multimodal Models — Cai et al. (2025) (arXiv:2502.00698, 2025)

What this evaluates

Probes human-like abstraction and visual reasoning capabilities in multimodal models across eight fine-grained paradigms, including logical operations, geometry, and spatial relationships. It measures how well models generalize to novel abstract patterns without relying on memorized visual features.

Datasets

• MM-IQ — total 2710; splits: test (2710), train (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Computed as the fraction of test items where the model's final answer matches the ground truth. Answers are extracted via regular expression matching from the model's full response; unmatched or missing answers are treated as incorrect.

Input / output format

Input: Zero-shot multimodal prompt containing an image and a question, identical across all evaluated models.

Output: A single free-form text response per instance, from which the final answer is extracted via regex.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        extracted = regex_extract_final_answer(pred)
        if extracted == gold:
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Regex-based answer extraction may fail if models use non-standard phrasing for the final answer.
• Few-shot prompting is explicitly excluded from the current benchmark, limiting reproducibility for that evaluation setting.
• Long chain-of-thought models are evaluated alongside short-CoT models, which may confound reasoning ability with generation length/format.

Evidence (verbatim from paper)

The evaluation process of LMMs consists of three steps: (1) response generation, (2) answer extraction, and (3) accuracy calculation. We extract the final answer using regular expression (regex) matching. For example, the final answer will be extracted from the response “The correct answer is A.” as “A”. If there is no valid answer in the model’s response, it will be considered incorrect.

Citation

@misc{cai2025mmiq,
  title={MM-IQ: Benchmarking Human-Like Abstraction and Reasoning in Multimodal Models},
  author={Cai et al. (2025)},
  year={2025},
  note={arXiv:2502.00698}
}

• arXiv: 2502.00698