Dreambench Eval

dreambench-eval

OmniGen: Unified Image Generation — Shitao Xiao et al. (2024) (arXiv:2409.11340, 2024)

What this evaluates

Evaluates subject-driven image generation by measuring how well the model follows text instructions and preserves the reference subject from the source image. It tests the model's ability to extract and reuse specific objects without fine-tuning.

Datasets

• DreamBench — total ?; splits: test (-1)

Metrics

• CLIP-T (primary) — range: [0, 1]
  • Cosine similarity between CLIP text embeddings of the instruction and the generated image.
• CLIP-I — range: [0, 1]
  • Cosine similarity between CLIP image embeddings of the reference subject image and the generated image.

Input / output format

Input: Reference image containing a subject, text instruction specifying the desired generation.

Output: Generated RGB image featuring the subject.

Scoring recipe

clip_model = load_clip()
clip_t = cosine_similarity(clip_model.encode_text(instruction), clip_model.encode_image(generated_img))
clip_i = cosine_similarity(clip_model.encode_image(reference_img), clip_model.encode_image(generated_img))

Common pitfalls

• CLIP-I may not accurately capture fine-grained subject identity preservation compared to face/object recognition models.
• The paper notes using a single reference image per object instead of fine-tuning, which differs from standard DreamBooth evaluation.

Evidence (verbatim from paper)

We evaluate the image editing on EMU-Edit[[58]] dataset and subject-driven generation capability on DreamBench[[56]]. We use CLIP-T to measure how well the model followed the instructions, while CLIP-I similarity scores measure the model’s ability to preserve elements from the source image.

Citation

@misc{xiao2024omnigen,
  title={OmniGen: Unified Image Generation},
  author={Shitao Xiao et al. (2024)},
  year={2024},
  note={arXiv:2409.11340}
}

• arXiv: 2409.11340