Refedit Eval

refedit-eval

RefEdit: A Benchmark and Method for Improving Instruction-based Image Editing Model on Referring Expressions — Pathiraja et al. (2025) (arXiv:2506.03448, 2025)

What this evaluates

Evaluates instruction-based image editing models on referring expressions, measuring how well they align edits with text instructions while preserving background and maintaining perceptual quality. It probes multi-object scene editing, background preservation, and the ability to handle complex spatial grounding without relying on CLIP.

Datasets

• RefEdit-Bench — total ?; splits: test (-1)
• PIE-Bench — total ?; splits: test (-1)

Metrics

• VIEScore (primary) — range: other
  • A training-free metric using GPT-4o to evaluate edited images. It comprises Semantic Consistency (SC), measuring alignment with the editing instruction, and Perceptual Quality (PQ), measuring authenticity and naturalness. The overall score is calculated as O = √(SC × PQ).
• Modified VIEScore — range: other
  • Same as VIEScore but incorporates ground truth masks to extract the region of interest before scoring, specifically used for PIE-Bench evaluations.
• Human Win Rate — range: percent
  • Percentage of times the model's output is preferred over baselines in A/B testing by human annotators on 400 randomly selected pairs.

Input / output format

Input: Original image and a text instruction (referring expression) specifying the desired edit.

Output: Edited image matching the instruction.

Scoring recipe

# VIEScore calculation
sc = gpt4o_prompt(image, instruction, template="SC")
pq = gpt4o_prompt(image, instruction, template="PQ")
overall = math.sqrt(sc * pq)

# Modified VIEScore (for PIE-Bench)
roi = extract_roi(original_image, ground_truth_mask)
sc_mod = gpt4o_prompt(roi, instruction, template="SC")
pq_mod = gpt4o_prompt(roi, instruction, template="PQ")
overall_mod = math.sqrt(sc_mod * pq_mod)

# Human Eval
win_rate = (count_preferred(model_output) / 400) * 100

Common pitfalls

• CLIP-based metrics fail on referring expressions due to poor spatial grounding, so VIEScore (GPT-4o) is required instead.
• Larger base models (e.g., SDXL, SD3) often overfit on complex referring expression tasks, performing worse than smaller SDv1.5 models.
• Ground truth masks are only used for the Modified VIEScore on PIE-Bench, not for the standard VIEScore on RefEdit-Bench.

Evidence (verbatim from paper)

However, as CLIP is not good at referring expression, we instead utilize VIEScore [[18]] as an alternative evaluation metric for both benchmarks, which is a training-free visual instruction-guided metric leveraging multimodal LLMs (MLLMs). It contains two metrics: Semantic Consistency (SC) and Perceptual Quality (PQ). SC determines the alignment of the edited image with the editing instruction and PQ determines the authenticity and the naturalness of the image. Following prompting templates from OmniEdit [[43]], we prompt GPT-4o [[29]] on SC and PQ. And we report the average performance. At last, we calculate the overall score as: O=\sqrt{SC\times PQ}.

Citation

@misc{pathiraja2025refedit,
  title={RefEdit: A Benchmark and Method for Improving Instruction-based Image Editing Model on Referring Expressions},
  author={Pathiraja et al. (2025)},
  year={2025},
  note={arXiv:2506.03448}
}

• arXiv: 2506.03448