Beep Korean Toxic Speech Eval

beep-korean-toxic-speech-eval

BEEP! Korean Corpus of Online News Comments for Toxic Speech Detection — Moon et al. (2020) (arXiv:2005.12503, 2020)

What this evaluates

This benchmark evaluates models' ability to detect social bias (gender and other types) and hate speech in Korean online news comments. It probes whether models can distinguish between hate speech, offensive language, and neutral comments, and whether incorporating bias labels improves hate speech detection.

Datasets

• BEEP! — total 9400; splits: test (-1); repo https://github.com/kocohub/korean-hate-speech

Metrics

• F1 (primary) — range: [0, 1]
  • Macro-F1 score computed across all classes for each task (binary/ternary bias identification and hate speech detection). It is the unweighted mean of the F1 score per class.

Input / output format

Input: Korean online news comment text.

Output: Classification label: for bias tasks, 'gender', 'other', or 'none' (ternary) or binary biased/not biased; for hate speech, 'hate', 'offensive', or 'none'.

Scoring recipe

def compute_macro_f1(preds, gold):
    classes = set(preds + gold)
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
    return sum(f1s) / len(f1s)

Common pitfalls

• CharCNN models heavily rely on surface-level lexical cues (e.g., gender pronouns) rather than semantic context, leading to false positives on neutral comments containing those words.
• The 'other' bias category is notoriously difficult to detect; models often default to predicting 'none' or 'gender', suggesting a two-step prediction strategy may be necessary.
• Term matching baselines combine 'hate' and 'offensive' into a single binary category, which is not directly comparable to the ternary hate speech task without adjustment.

Evidence (verbatim from paper)

Table 2 depicts F1 score of the three baselines and the term matching model. The results demonstrate that the models trained on our corpus have an advantage over the term matching method. Compared with the benchmarks, BERT achieves the best performance for all the three tasks: binary and ternary bias identification tasks, and hate speech detection.

Citation

@misc{moon2020beep,
  title={BEEP! Korean Corpus of Online News Comments for Toxic Speech Detection},
  author={Moon et al. (2020)},
  year={2020},
  note={arXiv:2005.12503}
}

• arXiv: 2005.12503