Clash Eval

clash-eval

CLASH: A Benchmark for Cross-Modal Contradiction Detection — Popordanoska et al. (2025) (arXiv:2511.19199, 2025)

What this evaluates

This benchmark probes multimodal large language models' ability to detect cross-modal contradictions between images and text. It evaluates whether models can identify inconsistencies when either modality contains errors or hallucinations, rather than assuming one modality is ground truth. The task reveals systematic modality biases and category-specific reasoning weaknesses.

Datasets

• CLASH — total 16289; splits: train (15000), test (1289)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly identified contradictory elements. For multiple-choice tasks, accuracy is calculated by selecting the 'Conflicting information – cannot answer' option. For open-ended tasks, accuracy is determined by whether the generated response correctly identifies the contradiction.

Input / output format

Input: An image (from MS COCO), a contradictory text caption, and a targeted question focusing on the conflicting element.

Output: Multiple-choice: selection of one of four options (image-grounded, text-grounded, plausible distractor, or 'Conflicting information – cannot answer'). Open-ended: free-form text response identifying the contradiction.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_ans in zip(predictions, gold):
        if pred == gold_ans:
            correct += 1
    return (correct / len(gold)) * 100

Common pitfalls

• Models often exhibit modality bias, defaulting to trusting either the image or the text rather than detecting the conflict.
• Open-ended generation tasks show significantly lower performance than multiple-choice, indicating models struggle to articulate contradictions without constrained options.
• Category-specific failures (e.g., confusing object vs. attribute contradictions) can skew aggregate accuracy if not analyzed separately.

Evidence (verbatim from paper)

revealing systematic modality biases and category-specific weaknesses—especially in open-source models—while demonstrating that targeted fine-tuning significantly improves conflict detection performance (e.g., LLaVA-1.5-7b improves from 0% to 77% accuracy).

Citation

@misc{popordanoska2025clash,
  title={CLASH: A Benchmark for Cross-Modal Contradiction Detection},
  author={Popordanoska et al. (2025)},
  year={2025},
  note={arXiv:2511.19199}
}

• arXiv: 2511.19199