Humanedit Eval

humanedit-eval

HumanEdit: A High-Quality Human-Rewarded Dataset for Instruction-based Image Editing — Bai et al. (2024) (arXiv:2412.04280, 2024)

What this evaluates

Evaluates instruction-based image editing models on their ability to modify source images according to textual prompts, with and without provided segmentation masks. It measures pixel-level fidelity, image quality, and text-image alignment across diverse editing categories such as add, remove, replace, action, counting, and relation.

Datasets

• HumanEdit — total 5751; splits: full (5751), core (-1), mask (-1), Add (-1), Action (-1), Counting (-1), Remove (-1), Relation (-1), Replace (-1)

Metrics

• L1 — range: other
  • Mean absolute pixel difference between the generated image and the ground truth image. Lower values indicate better pixel-level fidelity.
• L2 — range: other
  • Mean squared pixel difference between the generated image and the ground truth image. Lower values indicate better pixel-level fidelity.
• CLIP-I — range: [0, 1]
  • Cosine similarity between CLIP embeddings of the generated image and the ground truth image. Higher values indicate better image quality and visual consistency.
• DINO — range: [0, 1]
  • Cosine similarity between DINO embeddings of the generated image and the ground truth image. Higher values indicate better semantic-level image quality.
• CLIP-T (primary) — range: [0, 1]
  • Cosine similarity between CLIP embeddings of the local text description and the generated image. Higher values indicate better text-image alignment.

Input / output format

Input: Source image, textual editing instruction, and optionally a binary mask indicating the region to edit.

Output: Edited image matching the ground truth.

Scoring recipe

def compute_metrics(pred_img, gt_img, gt_text):
    l1 = np.mean(np.abs(pred_img - gt_img))
    l2 = np.mean((pred_img - gt_img) ** 2)
    clip_i = cosine_similarity(clip.encode(pred_img), clip.encode(gt_img))
    dino = cosine_similarity(dino.encode(pred_img), dino.encode(gt_img))
    clip_t = cosine_similarity(clip.encode(gt_text), clip.encode(pred_img))
    return {'L1': l1, 'L2': l2, 'CLIP-I': clip_i, 'DINO': dino, 'CLIP-T': clip_t}

Common pitfalls

• Comparisons between methods may be unfair due to differences in implementation and fine-tuning procedures.
• Pixel-level metrics (L1/L2) and semantic-level metrics (CLIP-I/DINO) can yield conflicting rankings, requiring careful interpretation.
• Mask-free and mask-provided settings operate under different constraints and should not be directly compared.

Evidence (verbatim from paper)

Follow the similar settings from previous works, we utilize L1 and L2 to measure the average pixel-level absolute difference between the generated image and ground truth image, and CLIP-I and DINO to measure the image quality with the cosine similarity between the generated image and reference ground truth image using their CLIP and DINO embeddings, and CLIP-T to measure the text-image alignment with the cosine similarity between local descriptions and generated images CLIP embeddings.

Citation

@misc{bai2024humanedit,
  title={HumanEdit: A High-Quality Human-Rewarded Dataset for Instruction-based Image Editing},
  author={Bai et al. (2024)},
  year={2024},
  note={arXiv:2412.04280}
}

• arXiv: 2412.04280