Magicbrush Eval

magicbrush-eval

UltraEdit: Instruction-based Fine-Grained Image Editing at Scale — Haozhe Zhao et al. (arXiv:2407.05282, 2024)

What this evaluates

Evaluates instruction-based image editing models on their ability to modify source images according to text instructions while preserving original content and style. It tests both single-turn and multi-turn editing capabilities against ground truth edited images.

Datasets

• MagicBrush — total ?; splits: test (-1)

Metrics

• L1 distance — range: other
  • Mean absolute pixel difference between the edited image and the ground truth image. Lower is better.
• L2 distance — range: other
  • Mean squared pixel difference between the edited image and the ground truth image. Lower is better.
• CLIP image similarity (primary) — range: [0, 1]
  • Cosine similarity between CLIP image embeddings of the edited image and the ground truth image. Higher is better.
• DINO similarity — range: [0, 1]
  • Cosine similarity between DINO image embeddings of the edited image and the ground truth image. Higher is better.

Input / output format

Input: Source image, editing instruction (caption), and optionally a region mask for fine-grained editing.

Output: Edited image.

Scoring recipe

import torch
def compute_metrics(edited_img, gt_img):
    clip_sim = cosine_similarity(clip_model.encode_image(edited_img), clip_model.encode_image(gt_img))
    dino_sim = cosine_similarity(dino_model.encode_image(edited_img), dino_model.encode_image(gt_img))
    l1 = torch.mean(torch.abs(edited_img - gt_img))
    l2 = torch.mean((edited_img - gt_img) ** 2)
    return l1, l2, clip_sim, dino_sim

Common pitfalls

• MagicBrush benchmark introduces biases towards its training set, causing models trained on it to overfit and lose generalization to other datasets.
• Evaluations differ significantly between single-turn and multi-turn settings; providing region mask input during evaluation boosts performance but may not reflect zero-shot region editing capability.

Evidence (verbatim from paper)

We evaluate the model trained on our dataset across two popular benchmarks: MagicBrush and Emu Edit Test. MaigicBrush benchmark evaluates the model by comparing the edited images with ground truth images and corresponding captions across different metrics. Following the MagicBrush, we chose the L1 distance, L2 distance, CLIP image similarity, and DINO similarity as metrics.

Citation

@misc{zhao2024ultraedit,
  title={UltraEdit: Instruction-based Fine-Grained Image Editing at Scale},
  author={Haozhe Zhao et al.},
  year={2024},
  note={arXiv:2407.05282}
}

• arXiv: 2407.05282