Dpo Preference Eval

dpo-preference-eval

Direct Preference Optimization: Your Language Model is Secretly a Reward Model — Rafael Rafailov et al. (arXiv:2305.18290, 2023)

What this evaluates

This evaluation protocol assesses a language model's ability to align with human preferences across open-ended text generation tasks. It measures how well the model optimizes a reward objective while staying close to a reference policy, and evaluates practical performance via pairwise win rates against baselines.

Datasets

• IMDb — total ?; splits: train (-1), test (-1)
• Reddit TL;DR — total ?; splits: train (-1), test (-1)
• Anthropic HH — total 170000; splits: test (-1)

Metrics

• win rate (primary) — range: [0, 1]
  • Computed as the fraction of pairwise comparisons where the model's generated output is preferred over a baseline output (e.g., reference summary or preferred response), judged by GPT-4 or human annotators.
• reward-KL frontier — range: other
  • A performance curve plotting expected reward against KL-divergence from the reference policy, used to evaluate the reward-regularization tradeoff in controlled settings.

Input / output format

Input: Text prompt: a movie review prefix, a Reddit forum post, or a single-turn human query.

Output: Generated text completion: a positive-sentiment continuation, a summary of the post, or a helpful/harmless response.

Scoring recipe

def compute_win_rate(predictions, baselines, evaluator):
    wins = 0
    total = 0
    for pred, base in zip(predictions, baselines):
        judge = evaluator.compare(pred, base)
        if judge == pred:
            wins += 1
        total += 1
    return wins / total if total > 0 else 0.0

Common pitfalls

• Relying on automatic metrics like ROUGE for summarization, which the paper explicitly notes are poorly correlated with human preferences.
• Failing to report win rates across multiple sampling temperatures, as model performance can degrade significantly at higher temperatures (0.0 to 1.0).
• Comparing win rates against different baseline policies without standardizing the reference (e.g., using ground truth summaries vs. preferred completions).

Evidence (verbatim from paper)

Our experiments use two different approaches to evaluation. In order to analyze the effectiveness of each algorithm in optimizing the constrained reward maximization objective, in the controlled sentiment generation setting we evaluate each algorithm by its frontier of achieved reward and KL-divergence from the reference policy; this frontier is computable because we have acccess to the ground-truth reward function (a sentiment classifier). However, in the real world, the ground truth reward function is not known; therefore, we evaluate algorithms with their win rate against a baseline policy, using GPT-4 as a proxy for human evaluation of summary quality and response helpfulness in the summarization and single-turn dialogue settings, respectively.

Citation

@misc{rafailov2023direct,
  title={Direct Preference Optimization: Your Language Model is Secretly a Reward Model},
  author={Rafael Rafailov et al.},
  year={2023},
  note={arXiv:2305.18290}
}

• arXiv: 2305.18290