qhjqhj00 m3cot-eval
M$^3$CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought — Chen et al. (2024) (arXiv:2405.16473, 2024)
What this evaluates
Evaluates vision-language models' ability to perform multi-step, multi-modal chain-of-thought reasoning across diverse domains like science, commonsense, and mathematics. It probes the model's capacity to integrate visual information with textual reasoning steps and produce accurate final answers under various prompting and fine-tuning setups.
Datasets
- M3CoT — total ?; splits: train (-1), test (-1); repo https://github.com/LightChen233/M3CoT
Metrics
accuracy(primary) — range: percent- Percentage of test instances where the model's final extracted answer exactly matches the ground truth label. Computed per domain and overall.
Input / output format
Input: An image paired with a question/prompt. Depending on the evaluation setup, the prompt may include few-shot examples, chain-of-thought instructions (e.g., 'Let's think step-by-step!'), or tool-use directives.
Output: A chain-of-thought reasoning trace followed by a final answer. The final answer is extracted using regular expressions.
Scoring recipe
def compute_accuracy(predictions, gold_answers):
correct = 0
for pred, gold in zip(predictions, gold_answers):
extracted = extract_final_answer(pred) # via regex as specified
if extracted == gold:
correct += 1
return (correct / len(gold_answers)) * 100
Common pitfalls
- Regex-based answer extraction may fail if models output answers in unexpected formats or include extra conversational text.
- Multi-step reasoning performance drops significantly with more steps, so evaluating only final accuracy without step-wise analysis can mask intermediate reasoning failures.
- Tool-usage and ICL setups can degrade performance if the model lacks interleaved image-text training or proper multi-modal planning capabilities.
Evidence (verbatim from paper)
In order to further understand the difference in model reasoning with different numbers of steps, we calculated the accuracy of different steps. As illustrated in Figure[7] (b), an increase in the number of reasoning steps is associated with a significant decline in the model’s performance. Following the settings of Kojima et al. (2022); Qin et al. (2023), we extract the final generated answer through regular expressions.
Citation
@misc{chen2024m3cot,
title={M$^3$CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought},
author={Chen et al. (2024)},
year={2024},
note={arXiv:2405.16473}
}
- arXiv: 2405.16473