Hate Speech Ordinal Eval

hate-speech-ordinal-eval

Constructing interval variables via faceted Rasch measurement and multitask deep learning: a hate speech application — Kennedy et al. (2020) (arXiv:2009.10277, 2020)

What this evaluates

Evaluates deep learning models' ability to predict continuous, interval-scaled hate speech scores from raw text comments. It benchmarks against existing APIs and transformer baselines using cross-validated error and correlation metrics.

Datasets

• Custom hate speech corpus (YouTube, Reddit, Twitter) — total ?; splits: train (42000), val (-1); repo https://github.com/ck37/coral-ordinal

Metrics

• RMSE (primary) — range: other
  • Root mean-squared error between predicted continuous hate scores and ground truth Rasch-calibrated scores.
• MAE — range: other
  • Mean absolute error between predicted and ground truth scores.
• Corr — range: [-1, 1]
  • Pearson linear correlation coefficient between predicted and ground truth scores.

Input / output format

Input: Raw text of a user comment.

Output: A single continuous float representing the predicted hate speech score.

Scoring recipe

preds = model.predict(comments)
rmse = np.sqrt(np.mean((preds - gold) ** 2))
mae = np.mean(np.abs(preds - gold))
corr = np.corrcoef(preds, gold)[0, 1]
return rmse, mae, corr

Common pitfalls

• The training data distribution is intentionally skewed during collection and does not reflect population-level hate speech prevalence.
• Twitter comments may be pre-filtered by the platform or API, leading to artificially low scores compared to YouTube/Reddit.
• Baseline models (Jigsaw) require linear OLS calibration to map binary probabilities to the continuous scale.

Evidence (verbatim from paper)

Direct prediction of the continuous hate score has currently achieved the lowest root mean-squared error (RMSE), although our proposed multitask networks that are transformed via IRT achieved comparable performance and slightly lower mean absolute error with the benefit of explainability.

Citation

@misc{kennedy2020constructing,
  title={Constructing interval variables via faceted Rasch measurement and multitask deep learning: a hate speech application},
  author={Kennedy et al. (2020)},
  year={2020},
  note={arXiv:2009.10277}
}

• arXiv: 2009.10277