Sass Eval

sass-eval

Critical Perspectives: A Benchmark Revealing Pitfalls in PerspectiveAPI — Piedras et al. (2023) (arXiv:2301.01874, 2023)

What this evaluates

This benchmark probes the ability of toxicity detection models to identify nuanced, adversarially crafted harmful content (e.g., gaslighting, manipulation, sarcasm) that mainstream tools often miss due to reliance on normative annotations and profanity cues.

Datasets

• SASS — total ?; splits: test (-1); repo https://github.com/lurosenb/sass

Metrics

• F1-Score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (Precision * Recall) / (Precision + Recall). Precision = TP/(TP+FP), Recall = TP/(TP+FN) for the positive (toxic) class.

Input / output format

Input: Raw text comment/string to be classified for toxicity.

Output: Binary label: 'toxic' or 'non-toxic' (derived from thresholding model/human toxicity scores at >0.5).

Scoring recipe

def compute_metrics(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
    return {'precision': precision, 'recall': recall, 'f1_score': f1}

Common pitfalls

• Thresholding continuous human toxicity scores at 0.5 to create binary ground truth may not align with model score distributions or practical toxicity definitions.
• Models may overfit to profanity detection rather than understanding malicious intent, leading to high false positives on non-toxic swear words.
• SASS is specifically adversarially crafted to exploit PERSPECTIVE's vulnerabilities, so performance may not generalize to standard toxicity benchmarks like TweetEval.

Evidence (verbatim from paper)

We showcase the utility of SASS by evaluating PERSPECTIVE and GPT-3 against the human baseline in a binary classification setting. We binarize the PERSPECTIVE and z-normalized human baseline toxicity scores by labeling scores $>0.5$ per comment as "toxic". We use these thresholded human labels as ground truth and evaluate PERSPECTIVE and GPT-3's performance on SASS in Table 3. We first observe that PERSPECTIVE performs very poorly on the binary task of toxicity classification on the SASS benchmark (Table 3, F1-Score $= 0.08$ ).

Citation

@misc{piedras2023critical,
  title={Critical Perspectives: A Benchmark Revealing Pitfalls in PerspectiveAPI},
  author={Piedras et al. (2023)},
  year={2023},
  note={arXiv:2301.01874}
}

• arXiv: 2301.01874