Peer Review Toxic Detection Eval

peer-review-toxic-detection-eval

Benchmark on Peer Review Toxic Detection: A Challenging Task with a New Dataset — Luo et al. (2025) (arXiv:2502.01676, 2025)

What this evaluates

This benchmark evaluates the ability of models to detect toxic sentences within academic peer reviews, where toxicity manifests as subtle emotive, rhetorical, or unconstructive language rather than overt abuse. It measures alignment with human judgments and assesses whether models can revise toxic sentences while preserving the original critique.

Datasets

• Peer Review Toxic Detection Dataset — total ?; splits: test (-1)

Metrics

• Cohen's Kappa (primary) — range: [-1, 1]
  • Measures inter-rater agreement between model predictions and human labels, adjusting for chance. κ = (p_o - p_e) / (1 - p_e), where p_o is observed agreement and p_e is expected agreement by chance.
• Precision — range: [0, 1]
  • Ratio of correctly predicted toxic sentences to all sentences predicted as toxic.
• Recall — range: [0, 1]
  • Ratio of correctly predicted toxic sentences to all actual toxic sentences.
• F1 — range: [0, 1]
  • Harmonic mean of Precision and Recall.
• Accuracy — range: [0, 1]
  • Ratio of correctly predicted sentences to total sentences.

Input / output format

Input: A single sentence from a peer review, optionally accompanied by prompt instructions (e.g., simple binary prompt, detailed toxicity definition, or toxicity summary).

Output: Binary label (0 for non-toxic, 1 for toxic) or text ('toxic'/'non toxic'), optionally followed by a confidence score (percentage).

Scoring recipe

def compute_metrics(preds, gold):
    tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
    tn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 0)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    acc = (tp + tn) / len(gold)
    p_e = ((tp + fp) / len(gold)) * ((tp + fn) / len(gold)) + ((fp + tn) / len(gold)) * ((fn + tn) / len(gold))
    kappa = (acc - p_e) / (1 - p_e) if (1 - p_e) > 0 else 0
    return {'precision': prec, 'recall': rec, 'f1': f1, 'accuracy': acc, 'kappa': kappa}

Common pitfalls

• General-domain toxic detection models fail on peer reviews because toxicity here lacks overt hostility and relies on subtle rhetorical or unconstructive cues.
• Open-source LLMs often fail to follow detailed instruction prompts, outputting unintended text instead of binary labels, requiring careful prompt engineering or filtering.
• Cohen's Kappa scores are highly sensitive to confidence thresholds; filtering by high confidence (>90-95%) artificially inflates alignment metrics by discarding ambiguous cases.

Evidence (verbatim from paper)

Because of the in-balanced labels in the testing set, we report Precision, Recall, F1, Accuracy scores. Furthermore, we report Cohen's Kappa between the human label and each model performance.

Citation

@misc{luo2025benchmark,
  title={Benchmark on Peer Review Toxic Detection: A Challenging Task with a New Dataset},
  author={Luo et al. (2025)},
  year={2025},
  note={arXiv:2502.01676}
}

• arXiv: 2502.01676