Alpaca Eval Lc Winrate Eval

alpaca-eval-lc-winrate-eval

Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignment — Takahashi et al. (2026) (arXiv:2604.04410, 2026)

What this evaluates

Evaluates the alignment quality of language models by measuring their win rate against a baseline on the AlpacaEval benchmark. It specifically uses length-controlled (LC) win rates to mitigate the known bias toward longer model outputs in standard auto-annotator evaluations.

Datasets

• alpaca_eval — total ?; splits: test (-1); HF tatsu-lab/alpaca_eval; repo https://github.com/tatsu-lab/alpaca_eval

Metrics

• AlpacaEval length-controlled (LC) win rate (primary) — range: percent
  • Win rate against a baseline model computed by an LLM-based auto-annotator, with a regression step applied to remove bias toward longer outputs.

Input / output format

Input: Text prompts from the AlpacaEval dataset.

Output: Model-generated text responses to each prompt.

Scoring recipe

def compute_lc_win_rate(prompts, model, baseline, annotator):
    preds = model.generate(prompts)
    base_preds = baseline.generate(prompts)
    scores = annotator.compare(preds, base_preds)
    raw_wr = mean(scores == 'win')
    lc_wr = regress_length_bias(raw_wr, preds, base_preds)
    return lc_wr

Common pitfalls

• Reporting raw win rates instead of length-controlled (LC) win rates, which ignores AlpacaEval's known bias toward longer generations.
• Using mismatched auto-annotator or baseline models (e.g., not weighted_alpaca_eval_gpt4_turbo and gpt-4-turbo-2024-04-09), which breaks comparability with reported baselines.
• Failing to average results over multiple random seeds, as the protocol requires repeating each experiment three times.

Evidence (verbatim from paper)

As the metric, we used AlpacaEval length-controlled (LC) win rates (Dubois et al., 2024). We generated responses from the aligned models on the alpaca_eval prompts, computed win rates against a baseline using an LLM-based auto-annotator, and reported LC win rates that regress out AlpacaEval’s bias toward longer outputs. We used gpt-4-turbo-2024-04-09 for the baseline and weighted_alpaca_eval_gpt4_turbo for the auto-annotator.

Citation

@misc{takahashi2026rdro,
  title={Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignment},
  author={Takahashi et al. (2026)},
  year={2026},
  note={arXiv:2604.04410}
}

• arXiv: 2604.04410