Alpacaeval Eval

alpacaeval-eval

Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators — Dubois et al. (2024) (arXiv:2404.04475, 2024)

What this evaluates

Evaluates LLM response quality via pairwise win rates against a baseline, while specifically probing the metric's susceptibility to length bias, gameability via verbosity prompting, and robustness to adversarial truncation.

Datasets

• AlpacaEval — total 805; splits: test (805)

Metrics

• Win rate (primary) — range: percent
  • Percentage of pairwise comparisons where the model's response is preferred over the baseline. Computed as 100% - winrate(baseline, model) ∈ [0%, 100%].
• Length gameability — range: percent
  • Normalized standard deviation of win rates across three verbosity prompts (concise, standard, verbose). Lower values indicate the metric is less sensitive to output length.
• Spearman correlation — range: [-1, 1]
  • Rank correlation between the benchmark's win rates and Chatbot Arena ELO ratings. Computed on benchmarks evaluating at least 25 models.

Input / output format

Input: Instruction, baseline response, and model response provided to an LLM judge for pairwise preference comparison.

Output: Pairwise preference (model wins, baseline wins, or tie), aggregated into a win rate percentage.

Scoring recipe

def compute_metrics(pairs, arena_elo):
    # 1. Win rate
    wins = sum(1 for p in pairs if p.model_preferred)
    win_rate = (wins / len(pairs)) * 100
    
    # 2. Length gameability
    win_rates = [compute_win_rate(prompt=p) for p in ['concise', 'standard', 'verbose']]
    gameability = normalized_std(win_rates)
    
    # 3. Spearman correlation with Chatbot Arena
    spearman_corr = spearmanr(benchmark_win_rates, arena_elo)
    return win_rate, gameability, spearman_corr

Common pitfalls

• Failing to control for output length can cause metrics to favor verbose but low-quality responses.
• Using Pearson correlation instead of Spearman for ELO ratings misrepresents the log-linear relationship.
• Stratification-based length control (length-balanced) can become unstable with few samples per stratum or vulnerable to truncation attacks.

Evidence (verbatim from paper)

We measure Spearman rather than Pearson correlation as probabilities are log-linearly correlated with ELO ratings, rather than linearly.

Citation

@misc{dubois2024lengthcontrolledalpacaeval,
  title={Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators},
  author={Dubois et al. (2024)},
  year={2024},
  note={arXiv:2404.04475}
}

• arXiv: 2404.04475