Cotomod Eval

cotomod-eval

Measuring and Improving Model-Moderator Collaboration using Uncertainty Estimation — Kivlichan et al. (2021) (arXiv:2107.04212, 2021)

What this evaluates

Evaluates how well AI models can assist human moderators in content moderation by prioritizing comments for review. It probes the model's ability to estimate uncertainty accurately and guide human review capacity to maximize collaborative accuracy and efficiency under constraints.

Datasets

• CoToMoD — total ?; splits: testing (-1), deployment (-1); repo http://github.com/google/uncertainty-baselines

Metrics

• OC-Acc (primary) — range: [0, 1]
  • Oracle-Model Collaborative Accuracy measures the combined system's accuracy when a human reviews the top-k comments ranked by the model's review score (toxicity or uncertainty), up to a capacity limit α. Accuracy is computed over all comments, assuming human review corrects errors.
• OC-AUC — range: [0, 1]
  • Area under the curve of collaborative accuracy as the human review capacity α varies continuously from 0% to 100%.
• Review Efficiency — range: percent
  • The percentage reduction in human review load required to achieve a target collaborative accuracy compared to a baseline review strategy.

Input / output format

Input: Raw text comments for toxicity classification.

Output: Predicted toxicity probability p(y|x), uncertainty score u(x), and binary gold label.

Scoring recipe

def compute_collaborative_metrics(preds, gold, scores, alpha):
    n = len(gold)
    k = int(alpha * n)
    top_k_idx = np.argsort(-scores)[:k]
    correct = 0
    for i in range(n):
        if i in top_k_idx:
            correct += (gold[i] == 1)  # Human perfect review
        else:
            correct += (preds[i] == gold[i])
    return correct / n

Common pitfalls

• Confusing uncertainty-based review (peaks at p=0.5) with toxicity-score-based review (peaks at p=1.0), leading to incorrect prioritization of comments.
• Ignoring class imbalance: Focal loss is required to handle easy negatives, otherwise uncertainty calibration degrades significantly on imbalanced datasets.
• Assuming higher uncertainty always indicates toxicity; uncertainty is highest at the decision boundary (p=0.5), not necessarily at extreme toxicity.

Evidence (verbatim from paper)

The paper introduces rigorous metrics—Oracle-Model Collaborative Accuracy (OC-Acc), OC-AUC, and Review Efficiency—to evaluate human-AI collaboration in content moderation under human capacity constraints. A large-scale study shows that uncertainty-based review strategies consistently outperform toxicity-score-based strategies across models and human review capacities, demonstrating that model uncertainty improves system efficiency and accuracy by prioritizing high-risk, low-confidence predictions for human review.

Citation

@misc{kivlichan2021measuring,
  title={Measuring and Improving Model-Moderator Collaboration using Uncertainty Estimation},
  author={Kivlichan et al. (2021)},
  year={2021},
  note={arXiv:2107.04212}
}

• arXiv: 2107.04212