Image Reward Eval

image-reward-eval

ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation — Xu et al. (2023) (arXiv:2304.05977, 2023)

What this evaluates

Evaluates a model's ability to predict human preferences for text-to-image generation by ranking pairs of images generated from the same text prompt. It measures alignment with human judgment on coherence, fidelity, and aesthetic quality.

Datasets

• ImageReward Test Set — total 6399; splits: test (6399); repo https://github.com/THUDM/ImageReward

Metrics

• Preference Accuracy (primary) — range: percent
  • Percentage of image pairs where the model's predicted preferred image matches the human-annotated preferred image for a given prompt. Calculated as correct predictions divided by total pairs.
• Recall@k / Filter@k — range: percent
  • Recall@k: proportion of prompts where the human-preferred image is ranked in the top-k by the model. Filter@k: proportion of prompts where the human-preferred image is among the top-k selected images from a set of 8. All scores are averaged per prompt.

Input / output format

Input: A text prompt and two (or k) images generated from that prompt.

Output: A scalar preference score for each image, or a ranked list of images.

Scoring recipe

def compute_preference_accuracy(predictions, gold):
    correct = 0
    for pred, gold_label in zip(predictions, gold):
        pred_label = 1 if pred[0] > pred[1] else 0
        if pred_label == gold_label:
            correct += 1
    return (correct / len(gold)) * 100

Common pitfalls

• The test set is curated by selecting prompts annotated by annotators with high agreement to researchers, which may not represent the full distribution of human preferences.
• Baseline models (CLIP, Aesthetic, BLIP) use fundamentally different scoring mechanisms (cosine similarity vs. learned predictors), so direct score comparison requires careful calibration or thresholding.
• The 'Filter' metric is mentioned but not explicitly defined in the text, leaving ambiguity about whether it refers to top-k selection or filtering based on a score threshold.

Evidence (verbatim from paper)

Preference accuracy is the correctness of a scorer choosing the same one from two different images of one prompt with a human. As Table 3 shows, our model outperforms all the baselines. The preference accuracy of ImageReward reaches up to 65.14%, which is 15.14% more than 50% (random), about twice as much as 7.76% (that of BLIP score).

Citation

@misc{xu2023imagereward,
  title={ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation},
  author={Xu et al. (2023)},
  year={2023},
  note={arXiv:2304.05977}
}

• arXiv: 2304.05977