Bradley Terry

bradley-terry

Confidence and Stability of Global and Pairwise Scores in NLP Evaluation — Levtsov et al. (2025) (arXiv:2507.01633, 2025)

What this evaluates

Evaluates the stability and reliability of global pointwise scores (accuracy, AUC, F1) versus pairwise Bradley-Terry rankings for ordering NLP models across classification and text generation tasks.

Datasets

• Jigsaw — total ?; splits: test (-1)
• SST-5 — total ?; splits: test (-1)
• CEval — total ?; splits: test (-1); repo https://github.com/aix-group/CEval-Counterfactual-Generation-Benchmark

Metrics

• Accuracy — range: [0, 1]
  • Proportion of correctly classified instances out of the total test set.
• AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the model's ability to distinguish between classes.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall, balancing false positives and false negatives.
• Bradley-Terry (primary) — range: other
  • A pairwise comparison model that estimates latent skill parameters for each model by maximizing the likelihood of observed pairwise preferences. Rankings are derived from these estimated parameters.
• Bradley-Terry binary — range: other
  • Binary variant of the Bradley-Terry model applied to pairwise comparisons.
• Spearman correlation — range: [-1, 1]
  • Rank-based correlation coefficient used to measure the monotonic relationship between different scoring methods' model rankings.

Input / output format

Input: Model predictions (decision-function values or probabilities) for classification tasks; generated text outputs for CEval. Ground-truth labels or majority-vote reconstructed labels are used for global scoring.

Output: Global score values per model (Accuracy, AUC, F1) and a ranked list of models derived from the Bradley-Terry pairwise comparison procedure.

Scoring recipe

def compute_global_scores(predictions, gold):
    acc = mean(predictions == gold)
    auc = compute_auc(predictions, gold)
    f1 = compute_f1(predictions, gold)
    return acc, auc, f1

def compute_bt_ranking(models, test_instances, m):
    n_samples = min(len(test_instances), 12 * m * log(m))
    sampled = sample_with_replacement(test_instances, n_samples)
    bt_scores = fit_bradley_terry(sampled, models)
    return sort_models_by(bt_scores, descending=True)

Common pitfalls

• Using all available test instances for pairwise comparisons instead of the recommended 12m log(m) sampling, which increases computational cost and may reduce ranking stability due to frequent ties.
• Reconstructing ground-truth labels via majority vote when official test labels are unavailable (as in Jigsaw), which can introduce bias and mask true model performance.
• Directly comparing global score magnitudes with pairwise Bradley-Terry rankings without accounting for their different scales, convergence rates, and sensitivity to rare high-impact errors.

Evidence (verbatim from paper)

For each test instance, we compared the outputs of $m$ different models in a pairwise fashion, yielding $\binom{m}{2}$ model pairs. For each pair, we then drew $12m\log(m)$ comparisons at random with replacement, or else used all available test instances if their count was smaller. Finally, we applied these sampled comparisons to build a Bradley–Terry ranking of the models.

Citation

@misc{levtsov2025confidence,
  title={Confidence and Stability of Global and Pairwise Scores in NLP Evaluation},
  author={Levtsov et al. (2025)},
  year={2025},
  note={arXiv:2507.01633}
}

• arXiv: 2507.01633