Multioff Hateful Meme Eval

multioff-hateful-meme-eval

MemeGraphs: Linking Memes to Knowledge Graphs — Kougia et al. (2023) (arXiv:2305.18391, 2023)

What this evaluates

Binary classification of memes as offensive or non-offensive. It probes a model's ability to detect hate speech in multimodal content by leveraging serialized scene graphs and knowledge graph entities alongside raw text.

Datasets

• MultiOFF — total 743; splits: train (445), val (149), test (149)

Metrics

• F1 score (offensive class) (primary) — range: [0, 1]
  • F1 score calculated specifically for the minority class (offensive). F1 = 2 * (Precision * Recall) / (Precision + Recall), where Precision = TP / (TP + FP) and Recall = TP / (TP + FN).

Input / output format

Input: Meme text concatenated with serialized scene graph triplets and/or knowledge base entity descriptions, separated by [SEP] tokens. Baseline models receive only the raw meme text.

Output: Probability score from a sigmoid activation, thresholded at 0.5 to yield a binary prediction (hateful vs. non-hateful).

Scoring recipe

def compute_f1_offensive(preds, gold):
    tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1

Common pitfalls

• The dataset is imbalanced (42% offensive), so accuracy is not reported; F1 for the minority class is the headline metric.
• Results are reported in two ways: average F1 over 20 random seeds (Table 1) vs. best checkpoint on validation set (Table 2). Comparing across tables requires matching the reporting convention.
• A fixed probability threshold of 0.5 is used for all models, which may not be optimal for the imbalanced test set.

Evidence (verbatim from paper)

For each model, we obtained predictions for the test set from all twenty differently initialized runs. We evaluated each prediction set by calculating the F1 score defining the minority class (offensive class) as positive. In Table 1, we report the average F1 score over the twenty runs for each method and the corresponding standard error.

Citation

@misc{kougia2023memegraphs,
  title={MemeGraphs: Linking Memes to Knowledge Graphs},
  author={Kougia et al. (2023)},
  year={2023},
  note={arXiv:2305.18391}
}

• arXiv: 2305.18391