Ethos Hate Speech Eval

ethos-hate-speech-eval

ETHOS: an Online Hate Speech Detection Dataset — Mollas et al. (2020) (arXiv:2006.08328, 2020)

What this evaluates

Evaluates the ability of NLP models to detect and classify hate speech in social media comments. It probes both binary hate/non-hate classification and multi-label categorization across specific demographic/identity-based hate categories. The benchmark emphasizes handling overlapping labels and class imbalance typical of real-world user-generated text.

Datasets

• ETHOS — total ?; splits: train (-1), test (-1); repo https://github.com/intelligence-csd-auth-gr/Ethos-Hate-Speech-Dataset.git

Metrics

• F1-score (macro) (primary) — range: percent
  • Macro-averaged F1 score computed across all hate speech categories, treating each label independently and averaging their F1 scores.
• Accuracy — range: percent
  • Standard classification accuracy calculated as the proportion of correctly predicted instances out of the total.
• Hamming Loss — range: [0, 1]
  • Symmetric difference between ground truth and predicted labels, normalized by the total number of labels.
• Subset Accuracy — range: percent
  • Exact match ratio where the predicted label set must perfectly match the ground truth label set for an instance to be counted as correct.

Input / output format

Input: Raw social media comments (YouTube/Reddit) preprocessed via lowercasing, contraction expansion, punctuation removal, and stemming/lemmatization (stemming/lemmatization explicitly skipped for Text-to-Sequence models).

Output: Binary label {0, 1} per category, derived by thresholding initial [0,1] discrete values at ≥0.5. Multi-label scope outputs a binary vector per instance.

Scoring recipe

# Binarize initial [0,1] values using 0.5 threshold
pred_bin = [1 if p >= 0.5 else 0 for p in pred]
gold_bin = [1 if g >= 0.5 else 0 for g in gold]

# Binary scope metrics
acc = accuracy_score(gold_bin, pred_bin)
f1_macro = f1_score(gold_bin, pred_bin, average='macro')

# Multi-label scope metrics
hamming = hamming_loss(gold_bin, pred_bin)
subset_acc = subset_accuracy(gold_bin, pred_bin)
precision_macro = precision_score(gold_bin, pred_bin, average='macro')
recall_macro = recall_score(gold_bin, pred_bin, average='macro')

Common pitfalls

• Dataset labels are initially discrete in [0,1] and must be binarized at a 0.5 threshold before evaluation; skipping this step yields incorrect scores.
• Evaluation relies on nested cross-validation for traditional ML and 10-fold CV for neural networks rather than a fixed held-out test set, which affects how results should be aggregated and compared.
• Stemming and lemmatization are explicitly skipped for Text-to-Sequence models (e.g., BERT, DistilBERT); applying them violates the stated protocol.

Evidence (verbatim from paper)

We chose accuracy and precision, recall and $F_{1}$-score with macro indication, and the confusion matrix as metrics. ... In the evaluation of MLL systems, a very common measure is the Hamming loss (symmetric difference between the ground truth labels and the predicted ones). Furthermore, subset accuracy (symmetric similarity), as well as precision, recall and $F_{1}$-score, are contained here (instance-based metrics).

Citation

@misc{mollas2020ethos,
  title={ETHOS: an Online Hate Speech Detection Dataset},
  author={Mollas et al. (2020)},
  year={2020},
  note={arXiv:2006.08328}
}

• arXiv: 2006.08328