Editreward Eval

editreward-eval

EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing — Wu et al. (2025) (arXiv:2509.26346, 2025)

What this evaluates

Evaluates the quality and human alignment of instruction-guided image editing models. It measures how well generated images match user instructions and visual realism, as well as how accurately models rank pairs of edited images according to human preferences.

Datasets

• ImagenHub — total ?; splits: test (-1)
• GenAI-Bench — total ?; splits: test (-1)
• AURORA-Bench — total ?; splits: test (-1)
• EditReward-Bench — total ?; splits: test (-1)
• GEdit-Bench — total ?; splits: test (-1)

Metrics

• Spearman rank correlation (primary) — range: [-1, 1]
  • Measures the monotonic relationship between predicted scores and human reference scores. Computed as the Pearson correlation between the rank-transformed predictions and gold scores.
• Pair-wise prediction accuracy (primary) — range: [0, 1]
  • The fraction of test instances where the model correctly predicts the preferred image in a given pair.
• GEdit-Bench scores (G_SQ, G_PQ, G_O) — range: [0, 10]
  • Scores on a 0-10 scale judged by GPT-4o: Semantic Consistency (G_SQ) for instruction fidelity, Perceptual Quality (G_PQ) for visual realism, and Overall Score (G_O) for overall quality.

Input / output format

Input: For point-wise benchmarks: original image, editing instruction, and edited image. For pair-wise benchmarks: original image, editing instruction, and two candidate edited images.

Output: Continuous scores (0-10 or correlation) or binary preference labels (which image is preferred).

Scoring recipe

def compute_accuracy(preds, golds):
    correct = sum(1 for p, g in zip(preds, golds) if p == g)
    return correct / len(golds)

def compute_spearman(preds, golds):
    return scipy.stats.spearmanr(preds, golds).correlation

Common pitfalls

• Confusing point-wise benchmarks (ImagenHub) which use Spearman correlation with pair-wise benchmarks (GenAI-Bench, AURORA-Bench) which use prediction accuracy.
• Failing to distinguish between the full test set and the 'Intersection' subset when reporting GEdit-Bench results.
• Comparing models trained on different backbones (Qwen2.5-VL-7B vs MiMo-VL-7B) without noting that performance scales with backbone strength.

Evidence (verbatim from paper)

For benchmarks with point-wise annotations like ImagenHub, we measure the Spearman rank correlation to assess alignment with human scores. For benchmarks with paired comparisons like GenAI-Bench and the pair-wise split of AURORA-Bench, we report the prediction accuracy.

Citation

@misc{wu2025editreward,
  title={EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing},
  author={Wu et al. (2025)},
  year={2025},
  note={arXiv:2509.26346}
}

• arXiv: 2509.26346