Instruction Editing Eval

instruction-editing-eval

Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated Data Refinement Using Contrastive Learning — Chen et al. (2025) (arXiv:2503.18406, 2025)

What this evaluates

Evaluates instruction-guided image editing models on their ability to modify an input image according to a text prompt. It measures semantic alignment with the prompt and visual fidelity to a ground-truth edit, plus human preference in pairwise comparisons.

Datasets

• MagicBrush (MagBr) — total ?; splits: test (-1)
• ZONE — total ?; splits: test (-1)

Metrics

• CLIP-T (primary) — range: [0, 1]
  • Cosine similarity between CLIP visual features of the generated image and CLIP text features of the target instruction: sim(CLIPvis(I^e'), CLIPtxt(p^e)).
• CLIP-I — range: [0, 1]
  • Cosine similarity between CLIP visual features of the generated image and CLIP visual features of the ground-truth edited image: sim(CLIPvis(I^e'), CLIPvis(I^e)).
• DINO-I — range: [0, 1]
  • Cosine similarity between DINOv2 visual features of the generated image and DINOv2 visual features of the ground-truth edited image: sim(DINOv2(I^e'), DINOv2(I^e)).

Input / output format

Input: Original image (I^o) and edit instruction (p^e)

Output: Edited image (I^e)'

Scoring recipe

def compute_clip_t(gen_img, target_prompt):
    gen_feat = clip_vision_model(gen_img)
    txt_feat = clip_text_model(target_prompt)
    return cosine_similarity(gen_feat, txt_feat)

def compute_visual_sim(gen_img, gt_img, model):
    gen_feat = model(gen_img)
    gt_feat = model(gt_img)
    return cosine_similarity(gen_feat, gt_feat)

Common pitfalls

• CLIP-T, CLIP-I, and DINO-I scores do not always correlate with human-perceived visual quality of edits.
• Ground-truth edited images (I^e) are unavailable for the ZONE benchmark, so CLIP-I and DINO-I cannot be computed for it.

Evidence (verbatim from paper)

We evaluate methods on two instruction-guided image-editing benchmarks: MagicBrush (MagBr) [32] and ZONE [11]. The former contains multi-turn edits where multiple instructions are used to edit one image iteratively, as opposed to single-turn edit where the image is edited once. We compare results quantitatively with the following metrics used in prior work [1, 11, 16, 32]: CLIP-T = sim(CLIPvis(I^e)', CLIPtxt(p^e)), CLIP-I = sim(CLIPvis((Ie)'), CLIPvis(Ie)), DINO-I = sim(DINOv2((Ie)'),DINOv2(Ie)) ... However, we note that as shown in Fig. 5 and in the supplemental material, these metrics do not always match the visual quality of the image edits, but are presented here for completeness.

Citation

@misc{chen2025instructclip,
  title={Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated Data Refinement Using Contrastive Learning},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2503.18406}
}

• arXiv: 2503.18406