Safe Speech Eval

safe-speech-eval

SafeSpeech: A Comprehensive and Interactive Tool for Analysing Sexist and Abusive Language in Conversations — Tan et al. (2025) (arXiv:2503.06534, 2025)

What this evaluates

Evaluates the capability of classifiers to detect sexist, abusive, offensive, and hate speech in conversational text across multiple granularity levels and established benchmarks. It probes fine-grained toxicity detection, cross-dataset generalization, and performance against strong supervised and LLM baselines.

Datasets

• EDOS (SemEval 2023) — total ?; splits: test (-1)
• OffensEval 2019 — total ?; splits: test (-1)
• AbusEval — total ?; splits: test (-1)
• HatEval — total ?; splits: test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Reported as absolute scores or percentage improvements over baselines.

Input / output format

Input: Textual messages or dialogue turns annotated for toxicity, sexism, hate speech, or abuse.

Output: Predicted labels corresponding to the evaluation subtask (binary detection, category-level classification, or fine-grained vector-level classification).

Scoring recipe

def compute_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == g and p == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• The paper reports F1 scores without specifying whether macro, micro, or weighted averaging is used, which is critical for multi-class and vector-level tasks like Subtask B and C.
• Out-of-domain evaluation is performed (training on EDOS, testing on OffensEval/AbusEval/HatEval), which may not reflect true generalization to unseen conversational contexts.
• Granularity levels differ across subtasks (binary vs. category vs. vector), making direct cross-subtask performance comparisons misleading without normalization.

Evidence (verbatim from paper)

The results demonstrate that M7-FE achieves top performance across all subtasks, improving over the winning system Zhou ([2023])* by 1% F1 on Subtask A and 4% F1 on Subtask C. On Subtask B, M7-FE matches the best reported performance while outperforming publicly available baselines such as ISEGURA/roberta-base_edos_b by 14% F1.

Citation

@misc{tan2025safespeech,
  title={SafeSpeech: A Comprehensive and Interactive Tool for Analysing Sexist and Abusive Language in Conversations},
  author={Tan et al. (2025)},
  year={2025},
  note={arXiv:2503.06534}
}

• arXiv: 2503.06534