Memotion Analysis Eval

memotion-analysis-eval

SemEval-2020 Task 8: Memotion Analysis -- The Visuo-Lingual Metaphor! — Chhavi Sharma et al. (2020) (arXiv:2008.03781, 2020)

What this evaluates

Evaluates multimodal understanding of internet memes by classifying five emotion categories (humor, sarcasm, offense, motivation) and overall sentiment from combined image and text inputs.

Datasets

• Memotion Analysis Dataset — total 9871; splits: train (6992), test (1879), trial (1000)

Metrics

• F1 score (primary) — range: [0, 1]
  • Macro-averaged F1 score computed independently for each of the five emotion/sentiment tasks. Calculated as the harmonic mean of precision and recall, averaged across all classes.

Input / output format

Input: Meme image and its embedded textual content (extracted via OCR and rectified by annotators).

Output: Five classification labels: Humour (NF/F/VF/H), Sarcasm (NS/G/TM/VT), Offense (NO/S/VO/HO), Motivation (NM/M), and Sentiment (VN/N/NEU/P/VP).

Scoring recipe

def compute_macro_f1(predictions, golds):
    f1_scores = []
    for task in ['humour', 'sarcasm', 'offense', 'motivation', 'sentiment']:
        preds_task = [p[task] for p in predictions]
        golds_task = [g[task] for g in golds]
        f1_scores.append(f1_score(golds_task, preds_task, average='macro'))
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• Subjective perception problem: annotations vary significantly across annotators due to personal interpretation of memes.
• Overlapping emotions: memes often belong to multiple categories simultaneously (e.g., both funny and sarcastic), complicating single-label assumptions.
• OCR inaccuracies: initial text extraction via Google Vision APIs often requires manual rectification by annotators.

Evidence (verbatim from paper)

Baseline F1 scores reach 0.35 (sentiment), 0.51 (humor), and 0.32 (intensity), indicating significant room for improvement in multimodal emotion modeling. For the challenge, 1K samples were provided as trial data, 6992 samples as training data while 1879 samples as test data.

Citation

@misc{sharma2020memotion,
  title={SemEval-2020 Task 8: Memotion Analysis -- The Visuo-Lingual Metaphor!},
  author={Chhavi Sharma et al. (2020)},
  year={2020},
  note={arXiv:2008.03781}
}

• arXiv: 2008.03781