Verite Eval

verite-eval

VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Bias — Papadopoulos et al. (2023) (arXiv:2304.14133, 2023)

What this evaluates

Evaluates multimodal misinformation detection models on real-world and synthetic image-caption pairs, specifically probing their ability to distinguish truthful content from out-of-context (OOC) and miscaptioned (MC) misinformation while measuring susceptibility to unimodal bias.

Datasets

• VERITE — total 1000; splits: test (1000); repo https://github.com/stevejpapad/image-text-verification
• COSMOS — total 1700; splits: test (1700)
• VMU-Twitter — total 1090; splits: test (1090); repo https://github.com/MKLab-ITI/image-verification-corpus

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly predicted labels out of the total number of test instances. Reported separately for binary (True vs Misinformation) and multiclass (True, OOC, MC) settings.
• percentage increase in accuracy ($\Delta\%) — range: percent
  • Measures unimodal bias by comparing unimodal model accuracy to multimodal model accuracy. Calculated as ((Acc_unimodal - Acc_multimodal) / Acc_multimodal) * 100. Negative values indicate multimodal advantage.
• Cohen's d effect size — range: other
  • Quantifies the standardized difference between unimodal and multimodal accuracy distributions. Positive values indicate unimodal bias favoring the unimodal model.

Input / output format

Input: Paired image and text (caption) inputs representing a single news or social media post.

Output: Classification label: 'True', 'OOC' (Out-of-Context), or 'MC' (Miscaptioned) for multiclass evaluation; or binary 'True' vs 'Misinformation' for VERITE-B and COSMOS evaluation.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

def compute_delta_percent(acc_unimodal, acc_multimodal):
    return ((acc_unimodal - acc_multimodal) / acc_multimodal) * 100

def compute_cohens_d(group1, group2):
    mean1, mean2 = np.mean(group1), np.mean(group2)
    var1, var2 = np.var(group1), np.var(group2)
    n1, n2 = len(group1), len(group2)
    pooled_std = np.sqrt(((n1-1)*var1 + (n2-1)*var2) / (n1+n2-2))
    return (mean1 - mean2) / pooled_std

Common pitfalls

• Prior works incorrectly used the test set for hyperparameter tuning on VMU-Twitter; the paper notes this flaw but follows it for comparability while also providing a corrected 90/10 train-val split.
• Small test set sizes for COSMOS and VERITE make out-of-distribution validation unreliable, forcing reliance on in-distribution generated data for hyperparameter tuning.
• Conflating OOC (out-of-context) and MC (miscaptioned) classes, which require different detection strategies and are often misaligned in unimodal bias analysis.

Evidence (verbatim from paper)

For evaluation, we report the accuracy score (image-only, text-only or multimodal) for binary classification on COSMOS and multiclass accuracy on VERITE. Moreover, we experiment with a binary version of VERITE (VERITE-B) where both “OOC” and “MC” pairs are combined into a single class denoting misinformation. Here, we report the accuracy for each pair of classes, namely “True vs OOC” and “True vs MC”. To evaluate the presence and magnitude of unimodal bias, we employ two metrics: the percentage increase in accuracy ($\Delta%) between a unimodal model and its multimodal counterpart, and Cohen’s d ($d$) effect size.

Citation

@misc{papadopoulos2023verite,
  title={VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Bias},
  author={Papadopoulos et al. (2023)},
  year={2023},
  note={arXiv:2304.14133}
}

• arXiv: 2304.14133