Emu Edit Eval

emu-edit-eval

OmniGen: Unified Image Generation — Shitao Xiao et al. (2024) (arXiv:2409.11340, 2024)

What this evaluates

Evaluates image editing capability by measuring instruction following (text similarity) and source image preservation (image similarity). It tests the model's ability to modify images based on textual instructions while maintaining relevant visual elements.

Datasets

• EMU-Edit — total ?; splits: test (-1)

Metrics

• CLIP-T (primary) — range: [0, 1]
  • Cosine similarity between the CLIP text encoder embeddings of the editing instruction and the generated image.
• CLIP-I — range: [0, 1]
  • Cosine similarity between the CLIP image encoder embeddings of the source image and the generated image.

Input / output format

Input: Source image, editing instruction (text), and optionally a target image or mask.

Output: Edited RGB image.

Scoring recipe

clip_model = load_clip()
clip_t = cosine_similarity(clip_model.encode_text(instruction), clip_model.encode_image(generated_img))
clip_i = cosine_similarity(clip_model.encode_image(source_img), clip_model.encode_image(generated_img))

Common pitfalls

• CLIP similarity scores can be inflated by background preservation rather than actual editing quality.
• Different baselines may use different preprocessing or guidance scales, affecting direct comparison.

Evidence (verbatim from paper)

We evaluate the image editing on EMU-Edit[[58]] dataset and subject-driven generation capability on DreamBench[[56]]. We use CLIP-T to measure how well the model followed the instructions, while CLIP-I similarity scores measure the model’s ability to preserve elements from the source image.

Citation

@misc{xiao2024omnigen,
  title={OmniGen: Unified Image Generation},
  author={Shitao Xiao et al. (2024)},
  year={2024},
  note={arXiv:2409.11340}
}

• arXiv: 2409.11340