Comt Eval

comt-eval

CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models — Cheng et al. (2024) (arXiv:2412.12932, 2024)

What this evaluates

Evaluates large vision-language models on chain-of-thought reasoning that requires generating both textual explanations and intermediate or final images. It probes the model's ability to perform four specific visual operations (creation, deletion, update, and selection) and align its multi-modal reasoning steps with ideal visual states.

Datasets

• CoMT — total ?; splits: test (-1); repo https://github.com/czhhzc/CoMT

Metrics

• F1 score (primary) — range: [0, 1]
  • Macro-averaged F1 score computed across the four visual operation tasks (creation, deletion, update, selection). Final answers are extracted from model outputs using regular expressions before comparison with gold labels.
• CLIPScore — range: [0, 1]
  • Average cosine similarity between CLIP image embeddings of model-generated reasoning images and pre-defined ideal rationale images, used to measure multi-modal alignment quality.

Input / output format

Input: An input image paired with a text prompt describing a visual reasoning task requiring one of four operations (creation, deletion, update, selection).

Output: A multi-modal reasoning chain containing textual steps and generated images, followed by a final answer.

Scoring recipe

def compute_comt_f1(model_output, gold_label):
    # Extract final answer using paper-specified regex
    final_answer = extract_regex(model_output)
    # Compute F1 against gold label
    f1 = compute_f1(final_answer, gold_label)
    return f1

# Average F1 across the four visual operation tasks
macro_f1 = np.mean([
    compute_comt_f1(out_creation, gold_creation),
    compute_comt_f1(out_deletion, gold_deletion),
    compute_comt_f1(out_update, gold_update),
    compute_comt_f1(out_selection, gold_selection)
])

Common pitfalls

• Pure text-only Chain-of-Thought prompts (e.g., 'Let's think step-by-step!') fail or degrade performance because models cannot execute visual logic in text alone.
• Extracting the final answer requires specific regular expressions; naive string matching or ignoring the multi-modal generation steps leads to incorrect F1 scores.
• Using more than four in-context demonstrations actually hurts performance due to token limits and context complexity, contrary to typical LLM expectations.

Evidence (verbatim from paper)

Following Qin et al. ([2023]) and Chen et al. ([2024b]), we extract the final generated answers using regular expressions. All LVLMs perform poorly on the CoMT. Despite Gemini achieving a 28.67% F1 score across four tasks, this performance is marginally better than the random baseline by 3.3%, indicating significant room for improvement.

Citation

@misc{cheng2024comt,
  title={CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models},
  author={Cheng et al. (2024)},
  year={2024},
  note={arXiv:2412.12932}
}

• arXiv: 2412.12932