Wac Eval

wac-eval

WAC: A Corpus of Wikipedia Conversations for Online Abuse Detection — Cécillon et al. (2020) (arXiv:2003.06190, 2020)

What this evaluates

Evaluates models on detecting online abuse (personal attacks, aggression, toxicity) in conversational contexts reconstructed from Wikipedia talk pages. It probes the ability to classify individual messages as abusive or non-abusive while leveraging or ignoring conversational structure depending on the method.

Datasets

• WAC (Wikipedia Conversations Corpus) — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/conversationai/wikidetox

Metrics

• Macro F-measure (primary) — range: [0, 1]
  • Harmonic mean of Precision and Recall, averaged across all classes (macro). F-measure = 2 * (Precision * Recall) / (Precision + Recall).

Input / output format

Input: Text of individual messages or conversation threads from Wikipedia talk pages.

Output: Binary classification label (abusive vs. non-abusive) or probability scores for each abuse category (Personal attack, Aggression, Toxicity).

Scoring recipe

def compute_macro_f1(predictions, gold_labels):
    precisions, recalls = [], []
    for label in ['personal_attack', 'aggression', 'toxicity']:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == label and g == label)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == label and g != label)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != label and g == label)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        precisions.append(prec)
        recalls.append(rec)
    f1s = [2 * p * r / (p + r) if (p + r) > 0 else 0 for p, r in zip(precisions, recalls)]
    return sum(f1s) / len(f1s)

Common pitfalls

• Evaluations across different papers often use inconsistent train/val/test splits, making direct comparison impossible.
• Binary conversion thresholds for external APIs (e.g., Perspective API) vary, leading to inflated or deflated precision/recall if not standardized.
• Graph-based methods require full conversation trees, while text-only methods only need isolated messages, causing unfair comparisons if context handling isn't controlled.

Evidence (verbatim from paper)

We propose a split into train $(60%)$ , development $(20%)$ and test $(20%)$ sets for each of the 3 datasets of WAC. This split was randomly generated, but is publicly available online. Using this split for all the methods ensure that all the results are obtained with the same data and so, are truly comparable. Additionally, we leave open the possibility to implement and add further metrics to the methods if needed, the tool being designed to ease the addition of new metrics. Different variants of the $F$ -Measure as well as the Area Under the ROC Curve are currently implemented, since they are the metrics mainly used by the methods listed in Table 2. Table 4: Macro Precision, Recall and $F$ -measure obtained by the 3 tested methods.

Citation

@misc{cecillon2020wac,
  title={WAC: A Corpus of Wikipedia Conversations for Online Abuse Detection},
  author={Cécillon et al. (2020)},
  year={2020},
  note={arXiv:2003.06190}
}

• arXiv: 2003.06190