Multimodal Cot Eval

multimodal-cot-eval

Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought — Cheng et al. (2025) (arXiv:2505.15510, 2025)

What this evaluates

Evaluates large vision-language models on multimodal chain-of-thought reasoning across mathematical, commonsense, visual grounding, and fine-grained identification tasks. It probes whether explicit intermediate visual representations improve cross-modal reasoning and transformer information flow.

Datasets

• IsoBench — total ?; splits: test (-1)
• MMVP — total ?; splits: test (-1)
• V*Bench — total ?; splits: test (-1)
• M3CoT-Commonsense — total ?; splits: test (-1)
• CoMT — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification/evaluation accuracy (correct predictions / total predictions). Not explicitly defined in the provided text; implied by standard benchmark reporting conventions.

Input / output format

Input: Multimodal inputs consisting of images and text prompts for large vision-language models.

Output: Text responses containing chain-of-thought reasoning and final answers.

Scoring recipe

def score(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip() == g.strip())
    return correct / len(gold)

Common pitfalls

• Temperature settings vary between [0, 2] for both open-source and proprietary models, which can significantly impact CoT generation quality and should be controlled.
• Different benchmarks measure distinct capabilities (math vs. commonsense vs. visual grounding); averaging scores across them without stratification is misleading.
• Hardware constraints (2 A6000 48G) may limit batch sizes or context lengths, potentially affecting throughput but not necessarily accuracy.

Evidence (verbatim from paper)

We select benchmarks from both math and commonsense categories. For the math tasks, we choose IsoBench involving tasks such as chess, math, graph, etc. For the commonsense tasks, we select datasets including MMVP, V*Bench, M3CoT-Commonsense, and CoMT, which assess the LVLMs’ capabilities such as visual grounding and object detection, fine-grained identification, and CoT reasoning.

Citation

@misc{cheng2025visualthoughts,
  title={Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought},
  author={Cheng et al. (2025)},
  year={2025},
  note={arXiv:2505.15510}
}

• arXiv: 2505.15510