Mme Cot Eval

mme-cot-eval

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency — Jiang et al. (2025) (arXiv:2502.09621, 2025)

What this evaluates

Evaluates the quality, robustness, and efficiency of Chain-of-Thought reasoning in Large Multimodal Models. It probes whether models can generate accurate intermediate reasoning steps, maintain performance consistency between direct and CoT prompting, and produce relevant, non-redundant reasoning traces.

Datasets

• MME-CoT — total ?; splits: test (-1)

Metrics

• F1 score (primary) — range: [0, 100]
  • Harmonic mean of precision and recall calculated over logical inference and image caption steps. Precision measures the accuracy of each reasoning step, while recall measures coverage of key inference conclusions or image observations.
• Robustness score — range: [0, 100]
  • Average of stability and efficacy. Stability measures the performance drop when switching from direct to CoT prompting on perception tasks. Efficacy measures accuracy on reasoning tasks under CoT prompting.
• Relevance rate — range: [0, 100]
  • Percentage of generated reasoning steps that are directly relevant to solving the given question, penalizing repetitive or exhaustive but irrelevant captions.
• Reflection quality — range: [0, 100]
  • Percentage of reflection steps that successfully correct mistakes or validate conclusions. Set to 100 for models incapable of reflection.

Input / output format

Input: Multimodal input consisting of an image and a question. Models are evaluated under two prompt conditions: a CoT prompt requesting step-by-step reasoning and a final answer, and a direct prompt requesting only the final answer.

Output: For CoT evaluation: a step-by-step intermediate reasoning process followed by a final answer. For direct evaluation: only the final answer.

Scoring recipe

def evaluate(predictions, gold, prompt_type):
    if prompt_type == 'direct':
        recall = judge_gpt4o_mini(predictions, gold, 'recall')
    else:
        precision = judge_gpt4o(predictions, gold, 'precision')
        recall = judge_gpt4o(predictions, gold, 'recall')
        relevance = judge_gpt4o(predictions, gold, 'relevance')
        reflection = judge_gpt4o(predictions, gold, 'reflection')
        f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0
    return {'F1': f1, 'Relevance': relevance, 'Reflection': reflection}
# Robustness = avg(stability, efficacy)
# Stability = direct_acc - cot_acc on perception tasks
# Efficacy = cot_acc on reasoning tasks

Common pitfalls

• Models often ignore the direct prompt and still generate CoT traces, artificially inflating robustness scores.
• Long CoT models may skip intermediate steps but arrive at the correct final answer, leading to misleadingly high recall/quality metrics.
• Reflection steps frequently fail or introduce interference, which can skew efficiency and quality scores if not properly penalized.

Evidence (verbatim from paper)

We report precision, recall, and relevance for both logical inference and image caption steps. For robustness, we provide the direct evaluation result on the perception and reasoning tasks, with either CoT or direct prompt. We employ the average value of the stability and efficacy as the final robustness metric. Notably, we define the reflection quality as 100 on models incapable of reflection.

Citation

@misc{jiang2025mme cot,
  title={MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency},
  author={Jiang et al. (2025)},
  year={2025},
  note={arXiv:2502.09621}
}

• arXiv: 2502.09621