Imgedit Bench Eval

imgedit-bench-eval

ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generation — Chen et al. (2025) (arXiv:2506.18095, 2025)

What this evaluates

Evaluates text-and-image-to-image editing capabilities, including addition, removal, replacement, motion change, style transfer, background change, object extraction, and hybrid edits. It tests the model's capacity to modify existing images according to natural language instructions while preserving unedited regions.

Datasets

• ImgEdit-Bench — total ?; splits: test (-1)

Metrics

• ImgEdit-Bench (primary) — range: other
  • Average score across 8 edit categories (Addition, Removal, Replacement, Motion Change, Style Transfer, Background Change, Object Extraction, Hybrid Edit).

Input / output format

Input: Source image paired with a text instruction specifying the edit type and target modification.

Output: Edited image.

Scoring recipe

scores = []
for img, instruction in dataset:
    edited_img = model.edit(img, instruction)
    score = judge.score(edited_img, img, instruction)
    scores.append(score)
return mean(scores)

Common pitfalls

• Edit categories often overlap, making isolated evaluation challenging.
• The exact scoring scale (e.g., 1-5) is not explicitly stated in the paper excerpt.

Evidence (verbatim from paper)

Table 3: Evaluation of text-and-image-to-image generation ability on ImgEdit-Bench.

Citation

@misc{chen2025sharegpt4oimage,
  title={ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generation},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2506.18095}
}

• arXiv: 2506.18095