Chatbot Arena Eval

chatbot-arena-eval

Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference — Chiang et al. (2024) (arXiv:2403.04132, 2024)

What this evaluates

Evaluates large language models by collecting human preference votes on pairwise responses to real-world prompts, then ranks them using Bradley-Terry models to measure alignment and real-world utility.

Datasets

• Chatbot Arena — total 213576; splits: all (213576)

Metrics

• BT coefficients (primary) — range: other
  • Models pairwise win probability as $P(A \succ B) = \frac{e^{\theta_A}}{e^{\theta_A} + e^{\theta_B}}$. Scores $\theta$ are estimated via maximum likelihood on collected votes, with confidence intervals computed using CLT or multiplicity correction.

Input / output format

Input: Pairwise model responses to a single user prompt.

Output: Human preference vote indicating which response is better (or a tie).

Scoring recipe

# Estimate BT scores via MLE on pairwise votes
scores = maximize_likelihood(votes, model=BradleyTerry)
# Compute confidence intervals for ranking stability
ci_lower, ci_upper = compute_clt_interval(scores, alpha=0.05)
# Rank models by estimated score
ranking = sort_models_by(scores, descending=True)
# Evaluate active sampling efficiency
sample_efficiency = compare_interval_widths(random_vs_adaptive_samples)

Common pitfalls

• Multiplicity correction for confidence intervals is technically required for formal ranking but makes intervals wider and more conservative, potentially altering rankings.
• Active sampling sample-efficiency gains are relative to a random baseline and depend heavily on the target precision threshold (e.g., 0.2 vs 0.3).
• Anomalous user detection thresholds (alpha) involve a direct trade-off between true positive and true negative rates.

Evidence (verbatim from paper)

For this experiment, we ran a replay of $T=213,576$ historical votes from our online platform and calculate the BT coefficients using our earlier-described estimation algorithm with confidence intervals; see Figure 5 for these intervals (with and without multiplicity correction; the formal notion of approximate ranking technically requires multiplicity correction, but it makes the intervals looser).

Citation

@misc{chiang2024chatbotarena,
  title={Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference},
  author={Chiang et al. (2024)},
  year={2024},
  note={arXiv:2403.04132}
}

• arXiv: 2403.04132