M3cotbench Eval

m3cotbench-eval

M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding — Jiang et al. (2026) (arXiv:2601.08758, 2026)

What this evaluates

This benchmark evaluates the Chain-of-Thought reasoning capabilities of multimodal large language models on medical image understanding tasks. It probes whether models can generate transparent, step-by-step diagnostic pathways that align with clinical ground truth, rather than just producing correct final answers.

Datasets

• M3CoTBench — total 1079; splits: test (-1)

Metrics

• F1 (primary) — range: percent
  • Harmonic mean of Precision and Recall calculated over aligned intermediate reasoning steps between the model output and ground truth. Values are reported as percentages.
• Impact — range: percent
  • Difference in accuracy between CoT prompting and direct prompting (Acc_step - Acc_direct), measuring whether step-by-step reasoning improves or degrades final diagnostic accuracy.
• Efficiency — range: other
  • Normalized score reflecting the computational or temporal cost of generating reasoning steps relative to direct answers, alongside raw latency in seconds.
• Consistency — range: percent
  • Percentage score measuring the similarity of reasoning paths generated across multiple runs or tasks for the same input, indicating stable diagnostic pathways.

Input / output format

Input: Medical image paired with a prompt instructing the model to either 'generate a step-by-step answer, including all intermediate reasoning steps, and provide the final answer at the end' (CoT) or 'directly provide the final answer without any additional output' (Direct).

Output: For CoT: a structured sequence of intermediate reasoning steps followed by a final diagnostic answer. For Direct: only the final diagnostic answer.

Scoring recipe

def compute_metrics(pred, gold):
    pred_steps = extract_reasoning_steps(pred)
    gold_steps = extract_reasoning_steps(gold)
    tp = len(set(pred_steps) & set(gold_steps))
    prec = tp / len(pred_steps) if pred_steps else 0
    rec = tp / len(gold_steps) if gold_steps else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    acc_direct = evaluate_final_answer(direct_pred, gold_answer)
    acc_step = evaluate_final_answer(step_pred, gold_answer)
    impact = acc_step - acc_direct
    paths = [extract_reasoning_steps(run) for run in multiple_runs]
    consistency = measure_path_similarity(paths)
    return f1, impact, consistency

Common pitfalls

• CoT prompting can degrade accuracy in medical image tasks by introducing hallucinated or misleading intermediate steps that distract from visual cues.
• Models may bypass step-by-step reasoning and output direct answers even when explicitly instructed to use CoT, artificially inflating efficiency but lowering alignment scores.
• Medical-specialized models often prioritize experience-driven conclusions over explicit step-by-step explanations, leading to lower CoT-GT alignment despite potentially correct final answers.

Evidence (verbatim from paper)

Consistency. Most models tend to generate similar reasoning steps when handling the same task, resulting in generally high path consistency scores. Most closed-source models achieve relatively high scores, benefiting from consistent generation processes and reasoning patterns. GPT-5 shows the lowest consistency because it often omits intermediate reasoning steps, producing incomplete chains.

Citation

@misc{jiang2026m3cotbench,
  title={M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding},
  author={Jiang et al. (2026)},
  year={2026},
  note={arXiv:2601.08758}
}

• arXiv: 2601.08758