Toxicity Perspectives Eval

toxicity-perspectives-eval

Designing Toxic Content Classification for a Diversity of Perspectives — Kumar et al. (2021) (arXiv:2106.04511, 2021)

What this evaluates

Evaluates how well automated toxicity classifiers align with diverse human perceptions of harmful content, specifically measuring how demographic background and personal harassment experiences influence toxicity judgments.

Datasets

• Toxicity Perspectives Dataset — total 107620; splits: full (107620)

Metrics

• interrater agreement (Cohen's kappa) (primary) — range: [-1, 1]
  • Cohen's kappa coefficient measuring the agreement between two independent raters on categorical ratings, adjusted for chance. Used to validate coding consistency and measure convergence of human ratings to a global average toxicity score.
• classification accuracy — range: [0, 1]
  • Fraction of comments correctly classified as toxic or not toxic when comparing model predictions to human labels, evaluated with fixed or personalized thresholds per demographic cohort.

Input / output format

Input: A social media comment text (sourced from Twitter, Reddit, or 4chan).

Output: Human ratings on a 5-point Likert scale ('Not at all toxic' to 'Extremely toxic'); model outputs a toxicity score between 0 and 1.

Scoring recipe

def compute_kappa(rater1_labels, rater2_labels):
    N = len(rater1_labels)
    Po = sum(1 for a, b in zip(rater1_labels, rater2_labels) if a == b) / N
    Pe = sum((count_r1[i] / N) * (count_r2[i] / N) for i in range(num_categories))
    return (Po - Pe) / (1 - Pe) if (1 - Pe) > 0 else 0

def compute_accuracy(predictions, human_labels, threshold):
    correct = sum(1 for p, h in zip(predictions, human_labels) if (p >= threshold) == (h >= threshold))
    return correct / len(predictions)

Common pitfalls

• Assuming a single global toxicity threshold applies to all users; the study shows thresholds must be personalized per demographic or experiential cohort.
• Treating the 5-point Likert scale as a single binary label without accounting for the specific cutoff ('Moderately toxic' or greater) used to define the positive class.
• Ignoring the stratified sampling bias: the dataset oversamples comments with API scores between 0.5 and 0.8 where human disagreement is highest.

Evidence (verbatim from paper)

Coding achieved an interrater agreement Cohen’s kappa κ>0.9 for all three variants, indicating strong agreement. In the event that a rater ascribed multiple themes to a single open ended response, we required both raters to select the same set of themes to constitute agreement.

Citation

@misc{kumar2021designing,
  title={Designing Toxic Content Classification for a Diversity of Perspectives},
  author={Kumar et al. (2021)},
  year={2021},
  note={arXiv:2106.04511}
}

• arXiv: 2106.04511