qhjqhj00 facebook-hateful-memes-eval
Labels or Input? Rethinking Augmentation in Multimodal Hate Detection — Singh et al. (2025) (arXiv:2508.11808, 2025)
What this evaluates
This benchmark evaluates multimodal hate speech detection by classifying image-caption pairs (memes) as hateful or non-hateful. It probes a model's ability to align visual and textual cues while resisting spurious correlations, particularly under different prompt structures and data augmentation strategies.
Datasets
- Facebook Hateful Memes — total 16940; splits: train (12900), val (1040), test (3000)
Metrics
accuracy— range: percent- Number of correctly classified instances divided by the total number of instances.
weighted-F1 score(primary) — range: percent- Harmonic mean of precision and recall, weighted by the number of true instances for each class (support).
Input / output format
Input: Paired image and natural language caption representing a meme.
Output: Binary classification label: 1 (hateful) or 0 (non-hateful).
Scoring recipe
def compute_metrics(preds, gold):
acc = sum(p == g for p, g in zip(preds, gold)) / len(gold)
f1_scores = []
for c in [0, 1]:
tp = sum(1 for p, g in zip(preds, gold) if p == c and g == c)
fp = sum(1 for p, g in zip(preds, gold) if p == c and g != c)
fn = sum(1 for p, g in zip(preds, gold) if p != c and g == c)
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
f1_scores.append(f1 * sum(1 for g in gold if g == c))
w_f1 = sum(f1_scores) / len(gold)
return acc, w_f1
Common pitfalls
- Using the original validation split for augmented experiments instead of the newly created 80:20 train-validation split.
- Relying solely on accuracy for imbalanced hate speech datasets without reporting weighted-F1, which masks poor minority-class performance.
- Mixing up binary and scaled label formats during prompt optimization, leading to inconsistent training objectives.
Evidence (verbatim from paper)
Our evaluation focuses on classification accuracy and weighted-F1 score, as standard in the hate classification benchmarks.
Citation
@misc{singh2025labelsorinput,
title={Labels or Input? Rethinking Augmentation in Multimodal Hate Detection},
author={Singh et al. (2025)},
year={2025},
note={arXiv:2508.11808}
}
- arXiv: 2508.11808