Modality Bias Eval

modality-bias-eval

Unveiling Modality Bias: Automated Sample-Specific Analysis for Multimodal Misinformation Benchmarks — Hehai Lin et al. (arXiv:2511.05883, 2025)

What this evaluates

This evaluation probes a model's ability to automatically detect and classify sample-specific modality bias in multimodal misinformation content. It measures how well automated quantification methods align with human judgment regarding whether a sample relies on image-only, text-only, or balanced modalities.

Datasets

• Fakeddit — total 680000; splits: eval (300)
• MMFakeBench — total ?; splits: eval (300)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of samples where the predicted bias type (Uni-image, Modality-balance, or Uni-text) matches the human annotation.
• F1 score — range: percent
  • Macro-averaged F1 score across the three bias classes compared to human annotations, computed per class and then averaged.

Input / output format

Input: Multimodal misinformation samples (image + text) randomly selected from benchmarks, processed through automated analysis frameworks (modality benefit, flow, causal effect) to predict bias type.

Output: Per-sample classification into one of three bias categories: Uni-image (UI), Modality-balance (MB), or Uni-text (UT). Aggregated metrics include predicted proportions per category, overall accuracy, and F1 score.

Scoring recipe

def compute_metrics(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    accuracy = (correct / len(gold_labels)) * 100
    f1_scores = []
    for cls in ['UI', 'MB', 'UT']:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != cls and g == cls)
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
        f1_scores.append(f1)
    f1_macro = (sum(f1_scores) / len(f1_scores)) * 100
    return accuracy, f1_macro

Common pitfalls

• The evaluation subset is small (300 samples per dataset) and randomly selected, which may not capture the full distribution of bias types present in the larger benchmarks.
• The paper reports both per-sample classification metrics (accuracy/F1) and dataset-level proportions, which can be conflated if not carefully distinguished.
• Human ground truth reliability is only summarized via Krippendorff's alpha in the appendix, making it difficult to assess annotation consistency without external lookup.

Evidence (verbatim from paper)

We report the predicted proportions of each modality bias type and the percentage that aligns with human judgment. For example, $0.78[85.53]$ denotes that multi-view analysis classifies 0.78 of the samples as modality-balance, and among these samples, 85.53% of the results are consistent with human judgment. We also report the overall accuracy and F1 score.

Citation

@misc{lin2025unveiling,
  title={Unveiling Modality Bias: Automated Sample-Specific Analysis for Multimodal Misinformation Benchmarks},
  author={Hehai Lin et al.},
  year={2025},
  note={arXiv:2511.05883}
}

• arXiv: 2511.05883