Hateful Memes Eval

hateful-memes-eval

Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learning — Jingbiao Mei et al. (2023) (arXiv:2311.08110, 2023)

What this evaluates

Evaluates multimodal models' ability to detect hateful or harmful memes by analyzing the alignment between image and text content. It probes robustness against visual and textual confounders that appear benign individually but become harmful when combined.

Datasets

• HatefulMemes — total ?; splits: train (-1), dev (-1), test (-1)
• HarMeme — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Computed by plotting the true positive rate against the false positive rate at various classification thresholds and calculating the area under the curve.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified instances (hateful vs. benign) out of the total number of instances.

Input / output format

Input: Multimodal input consisting of a meme image and its associated text/caption.

Output: Binary classification label: 'hateful' (combining very harmful and partially harmful) or 'benign' (harmless).

Scoring recipe

def compute_metrics(predictions, labels):
    acc = np.mean(predictions.round() == labels)
    auc = roc_auc_score(labels, predictions)
    return {'AUC': auc, 'Accuracy': acc}

Common pitfalls

• HarMeme originally has 3 classes (very harmful, partially harmful, harmless); the paper merges the first two into 'hateful', which may cause confusion if comparing to other works using the original split.
• Several baseline models (e.g., Flamingo-80B, HateCLIPper) only report AUC scores, making direct accuracy comparisons impossible for those baselines.
• The KNN majority voting classifier does not output raw logits, which artificially lowers its AUC score compared to logistic regression trained on the same embeddings.

Evidence (verbatim from paper)

To make a fair comparison, we adopt the evaluation metrics commonly used in existing hateful meme classification studies Kumar and Nandakumar (2022); Cao et al. (2022); Kiela et al. (2021): Area Under the Receiver Operating Characteristic Curve (AUC) and Accuracy (Acc). We train the system on the training split, develop them on the development splits and report the final results on the test set.

Citation

@misc{mei2023improving,
  title={Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learning},
  author={Jingbiao Mei et al. (2023)},
  year={2023},
  note={arXiv:2311.08110}
}

• arXiv: 2311.08110