Dialogue Safety Robustness Eval

dialogue-safety-robustness-eval

Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack — Dinan et al. (2019) (arXiv:1908.06083, 2019)

What this evaluates

Evaluates the robustness of offensive language detection models against adversarial human attacks in single-turn and multi-turn dialogue contexts. It measures classifier resilience when exposed to iterative, context-aware attacks designed to evade safety filters.

Datasets

• Wikipedia Toxic Comments — total ?; splits: test (-1)

Metrics

• Weighted-F1 (primary) — range: [0, 1]
  • Weighted average of F1 scores across classes (SAFE and OFFENSIVE), weighted by the number of true instances per class.
• F1 (OFFENSIVE) — range: [0, 1]
  • Harmonic mean of precision and recall calculated specifically for the OFFENSIVE class.

Input / output format

Input: Dialogue context (single-turn or 4-turn multi-turn) containing a target utterance to be classified.

Output: Binary label: SAFE or OFFENSIVE.

Scoring recipe

def compute_metrics(preds, gold):
    # Calculate TP, FP, FN for each class
    prec = TP / (TP + FP)
    rec = TP / (TP + FN)
    f1 = 2 * prec * rec / (prec + rec)
    # Weighted-F1: average F1 weighted by class support
    weighted_f1 = sum(f1[c] * support[c] for c in classes) / total_samples
    return weighted_f1, f1['OFFENSIVE']

Common pitfalls

• Adversarial evaluation is round-dependent; performance varies significantly depending on which attack round (1, 2, or 3) generated the test data.
• Multi-turn evaluation uses a fixed 4-turn context window, which may not generalize to longer or more complex conversations.
• Human evaluation scores (0-5) measure attack success/quality but the exact annotation rubric is not specified in this section.

Evidence (verbatim from paper)

We report F1, precision, and recall for the OFFENSIVE class, as well as weighted-F1 for models $S_{i}$ and $A_{i}$ on the single-turn standard and adversarial tasks in Table 13.

Citation

@misc{dinan2019builditbreakit,
  title={Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack},
  author={Dinan et al. (2019)},
  year={2019},
  note={arXiv:1908.06083}
}

• arXiv: 1908.06083