Memotion 2.0 Eval

memotion-2.0-eval

BLUE at Memotion 2.0 2022: You have my Image, my Text and my Transformer — Bucur et al. (2022) (arXiv:2202.07543, 2022)

What this evaluates

Evaluates models on classifying social media memes for sentiment, emotion intensity (humour, sarcasm, offensiveness), and motivation. It probes the ability of text-only and multi-modal architectures to perform both binary and ordinal/multi-class classification across multiple related subtasks.

Datasets

• Memotion 2.0 — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• weighted F1 (primary) — range: [0, 1]
  • The weighted average of per-class F1 scores, where each class's F1 is multiplied by its support (number of true instances) and divided by the total number of instances.

Input / output format

Input: Meme images and associated text captions, optionally processed into embeddings (e.g., CLIP features, image features) and text tokens.

Output: Predicted class labels for each emotion subtask (Sentiment, Humour, Sarcasm, Offensive, Motivation) in either binary or ordinal/multi-class format depending on the task definition.

Scoring recipe

def compute_weighted_f1(y_true, y_pred, classes):
    total_weight = 0
    weighted_f1_sum = 0
    for c in classes:
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
        support = sum(1 for t in y_true if t == c)
        weighted_f1_sum += f1 * support
        total_weight += support
    return weighted_f1_sum / total_weight if total_weight > 0 else 0

Common pitfalls

• Task B uses binary classification while Task C uses ordinal/multi-class intensity scales; models must map non-zero intensity predictions to the positive class for Task B if trained on Task C.
• Multi-task learning benefits vary by emotion; joint training improves humour, sarcasm, and offensiveness but may slightly hurt sentiment classification compared to single-task models.
• Modality ablation shows small differences; adding CLIP features improves results marginally, but the best submission combines all available modalities.

Evidence (verbatim from paper)

In order to measure the benefits of multi-task learning for classifying emotion intensities, we performed an ablation study by comparing the weighted F1 scores computed for each emotion intensity predictions made by models trained in two settings.

Citation

@misc{bucur2022blue,
  title={BLUE at Memotion 2.0 2022: You have my Image, my Text and my Transformer},
  author={Bucur et al. (2022)},
  year={2022},
  note={arXiv:2202.07543}
}

• arXiv: 2202.07543