Conda Eval

conda-eval

CONDA: a CONtextual Dual-Annotated dataset for in-game toxicity understanding and detection — Weld et al. (2021) (arXiv:2106.06213, 2021)

What this evaluates

Evaluates in-game toxicity detection using a dual-level NLU framework that jointly predicts utterance-level toxicity intent and token-level semantic slots. It probes a model's ability to understand contextual, game-specific language and distinguish between explicit, implicit, and action-based toxicity.

Datasets

• CONDA — total 44869; splits: train (26921), val (8974), test (8974)

Metrics

• UCA (primary) — range: [0, 1]
  • Utterance Classification Accuracy measures sentence-level classification performance based on the ratio of correctly predicted utterances to the total number of utterances.
• U-F1 — range: [0, 1]
  • Utterance F1 score calculates the F1 score for each utterance intent class (Explicit, Implicit, Action, Other).
• T-F1 — range: [0, 1]
  • Token F1 score focuses on prediction performance for slot tokens, calculating an F1 for each class and a token-based micro-averaged F1 over all classes excluding label O.
• JSA — range: [0, 1]
  • Joint Semantic Accuracy measures overall prediction performance over the semantic hierarchy. An utterance is deemed correctly analysed only if both utterance-level and all token-level labels including O are correctly predicted.

Input / output format

Input: Tokenised utterances with punctuation removed.

Output: Joint predictions of utterance-level intent labels (Explicit, Implicit, Action, Other) and token-level slot labels (Toxicity, Slang, Character, Dota-specific, Pronoun, Other).

Scoring recipe

def evaluate(preds, gold):
    uca = sum(1 for (ip, sp), (ig, sg) in zip(preds, gold) if ip == ig) / len(preds)
    u_f1 = {}
    for cls in ['E', 'I', 'A', 'O']:
        tp = sum(1 for (ip, _), (ig, _) in zip(preds, gold) if ip == cls and ig == cls)
        fp = sum(1 for (ip, _), (ig, _) in zip(preds, gold) if ip == cls and ig != cls)
        fn = sum(1 for (ip, _), (ig, _) in zip(preds, gold) if ip != cls and ig == cls)
        p = tp/(tp+fp) if tp+fp else 0; r = tp/(tp+fn) if tp+fn else 0
        u_f1[cls] = 2*p*r/(p+r) if p+r else 0
    t_f1 = {}
    for cls in ['T', 'S', 'C', 'D', 'P']:
        # identical TP/FP/FN logic applied to token-level predictions
        ...
    jsa = sum(1 for (ip, sp), (ig, sg) in zip(preds, gold) if ip == ig and sp == sg) / len(preds)
    return {'UCA': uca, 'U-F1': u_f1, 'T-F1': t_f1, 'JSA': jsa}

Common pitfalls

• Class O (Other) dominates the dataset numerically, which can artificially inflate its F1 score and mask performance on minority classes.
• Implicit toxicity (Class I) relies heavily on contextual understanding, making it significantly harder to detect than explicit toxicity.
• Game-specific tokens (Class D) exhibit flexible and variant forms, increasing detection difficulty and lowering T-F1 scores.

Evidence (verbatim from paper)

We split the data into train/validation/test sets in the proportions of 0.6/0.2/0.2, or in samples 26,921/8,974/8,974. The data passed to the models is the tokenised utterances with punctuation removed, and for training the slot and intent labels. We propose to use the following four metrics for conducting a multi-aspect evaluation... UCA: Utterance Classification Accuracy measures the sentence-level classification performance based on the ratio of the number of correctly predicted utterance to the total number of utterances. U-F1: Utterance F1 score calculates the F1 score for each utterance class. T-F1: Token F1 score focuses on the prediction performance for slot tokens and calculates an F1 for each class and the token-based micro-averaged F1 score over all classes excluding label O. JSA: Joint Semantic Accuracy measures the overall prediction performance over the semantic hierarchy. An utterance is deemed correctly analysed only if both utterance-level and all the token-level labels including Os are correctly predicted.

Citation

@misc{weld2021conda,
  title={CONDA: a CONtextual Dual-Annotated dataset for in-game toxicity understanding and detection},
  author={Weld et al. (2021)},
  year={2021},
  note={arXiv:2106.06213}
}

• arXiv: 2106.06213