qhjqhj00 semeval2021task6-st2
SemEval-2021 Task 6: Detection of Persuasion Techniques in Texts and Images — Dimitrov et al. (2021) (arXiv:2105.09284, 2021)
What this evaluates
Identifies and classifies spans of persuasion techniques within unimodal text using sequence tagging. It probes the model's ability to localize and categorize persuasive spans at the token level.
Datasets
- SemEval-2021 Task 6 Subtask 2 — total ?; splits: test (-1)
Metrics
F1(primary) — range: [0, 1]- Micro-averaged F1 score for sequence tagging of technique spans.
Input / output format
Input: Raw text snippet.
Output: Sequence of token-level labels indicating the start and end of persuasion technique spans.
Scoring recipe
tp = fp = fn = 0
for p, g in zip(predictions, golds):
if p == g: tp += 1
elif p in golds: fp += 1
elif g in predictions: fn += 1
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
Common pitfalls
- Handling BPE subword boundaries correctly is crucial; classes should be assigned as the union of subword predictions.
- Data sparseness and imbalance require careful loss design or augmentation.
Evidence (verbatim from paper)
Table 5 shows the evaluation results. We report our random baseline, which is based on the random selection of spans with random lengths and a random assignment of labels... Table 5: Results for Subtask 2. The systems are ordered by the official score: F1-micro.
Citation
@misc{dimitrov2021semevaltask6,
title={SemEval-2021 Task 6: Detection of Persuasion Techniques in Texts and Images},
author={Dimitrov et al. (2021)},
year={2021},
note={arXiv:2105.09284}
}
- arXiv: 2105.09284