qhjqhj00 offenseval-2020-eval
UPB at SemEval-2020 Task 12: Multilingual Offensive Language Detection on Social Media by Fine-tuning a Variety of BERT-based Models — Tanase et al. (2020) (arXiv:2010.13609, 2020)
What this evaluates
Binary classification of offensive versus non-offensive language in social media text across five languages (English, Arabic, Danish, Greek, Turkish). It probes multilingual transformer models' ability to detect hate speech and offensive content in both high-resource and low-resource settings.
Datasets
- SemEval-2020 Task 12 (Offenseval) — total ?; splits: train (-1), val (-1), test (-1)
Metrics
F1-score(primary) — range: [0, 1] (reported as percent)- Harmonic mean of precision and recall: 2 * (Precision * Recall) / (Precision + Recall). Computed per language on the validation set for model selection and on the hidden test set for leaderboard ranking.
Input / output format
Input: Raw social media text (tweets) in one of five languages (English, Arabic, Danish, Greek, Turkish).
Output: Binary label indicating whether the text is offensive or non-offensive.
Scoring recipe
def compute_f1(predictions, gold_labels):
tp = sum(1 for p, g in zip(predictions, gold_labels) if p == 1 and g == 1)
fp = sum(1 for p, g in zip(predictions, gold_labels) if p == 1 and g == 0)
fn = sum(1 for p, g in zip(predictions, gold_labels) if p == 0 and g == 1)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
return 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0
Common pitfalls
- Using English-specific preprocessing (e.g., n-gram TFIDF) on non-English data severely degrades performance.
- Translating non-English tweets to English before classification yields poor results compared to using multilingual models.
- Model selection is based on validation set F1-score, which may not perfectly correlate with hidden test set performance.
Evidence (verbatim from paper)
The reported metrics are computed for the Offenseval 2020 language-specific validation sets as described in Section 4. For each language, the highest validation set F1-score is highlighted, meaning that the corresponding model was selected and employed for predicting the language-specific competition test data in our final submission.
Citation
@misc{tanase2020upb,
title={UPB at SemEval-2020 Task 12: Multilingual Offensive Language Detection on Social Media by Fine-tuning a Variety of BERT-based Models},
author={Tanase et al. (2020)},
year={2020},
note={arXiv:2010.13609}
}
- arXiv: 2010.13609