Pairwise Preference Prediction Accuracy

pairwise-preference-prediction-accuracy

Alignment of Diffusion Models: Fundamentals, Challenges, and Future — Liu et al. (2024) (arXiv:2409.07253, 2024)

What this evaluates

Measures how well a reward model predicts human preference by comparing its scoring of image pairs against ground-truth human choices. It probes the model's ability to generalize alignment signals to unseen prompts and image distributions.

Datasets

• Pick-a-Pic v1 — total ?; splits: test (-1)

Metrics

• pairwise preference prediction accuracy (primary) — range: percent
  • Ratio of test cases where the reward model assigns a higher score to the image-text pair that was preferred by humans over the alternative pair generated from the same prompt.

Input / output format

Input: A text prompt and a pair of generated images (or image-text pairs) to be scored by the reward model, along with the ground-truth human preference label for the pair.

Output: A scalar accuracy value (percentage or 0-1 score) indicating the fraction of correctly predicted human preferences.

Scoring recipe

correct = 0
for prompt, img_a, img_b, human_pref in test_set:
    score_a = reward_model(prompt, img_a)
    score_b = reward_model(prompt, img_b)
    if (human_pref == 'A' and score_a > score_b) or (human_pref == 'B' and score_b > score_a):
        correct += 1
accuracy = correct / len(test_set)

Common pitfalls

• Models often overfit to the specific data distribution of their training benchmark, leading to high accuracy on native datasets but poor generalization to out-of-distribution prompts.
• Using a reward model to evaluate a policy trained with that same model can cause circular reasoning and inflated scores ('evaluation hacking').

Evidence (verbatim from paper)

To evaluate the performance of reward models in predicting human preference, the classical metric used is pairwise preference prediction accuracy. To calculate this accuracy, the reward model is first used to score a pair of images with the same prompt. The accuracy is then determined by the ratio of cases where the reward model assigns a higher score to the image-text pair preferred by humans on the test set.

Citation

@misc{liu2024alignmentdiffusion,
  title={Alignment of Diffusion Models: Fundamentals, Challenges, and Future},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2409.07253}
}

• arXiv: 2409.07253