qhjqhj00 semeval2021task6-st3
SemEval-2021 Task 6: Detection of Persuasion Techniques in Texts and Images — Dimitrov et al. (2021) (arXiv:2105.09284, 2021)
What this evaluates
Detects persuasion techniques in multimodal memes by jointly analyzing text and image content. It probes cross-modal alignment and interaction modeling for persuasive content detection.
Datasets
- SemEval-2021 Task 6 Subtask 3 — total ?; splits: test (-1)
Metrics
F1-Micro(primary) — range: [0, 1]- Micro-averaged F1 score for multimodal persuasion technique detection.
Input / output format
Input: Paired text and image (meme).
Output: Multilabel classification of persuasion techniques present in the meme.
Scoring recipe
tp = fp = fn = 0
for p, g in zip(predictions, golds):
for label in all_labels:
if label in p and label in g: tp += 1
elif label in p and label not in g: fp += 1
elif label not in p and label in g: fn += 1
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
Common pitfalls
- Modeling modality interaction is more important than simple feature fusion.
- Text and image representations often use different architectures (e.g., BERT for text, ResNet for images) requiring careful alignment.
Evidence (verbatim from paper)
The evaluation results are shown in Table 3... The systems are ordered by the official score: F1-micro. Subtask 3 (Multimodal: Memes)... This is a very rich and very interesting table. We can see that transformers were quite popular for text representation... For the visual modality, the most common representations were variants of ResNet...
Citation
@misc{dimitrov2021semevaltask6,
title={SemEval-2021 Task 6: Detection of Persuasion Techniques in Texts and Images},
author={Dimitrov et al. (2021)},
year={2021},
note={arXiv:2105.09284}
}
- arXiv: 2105.09284