Gedit Imgedit Bench Eval

gedit-imgedit-bench-eval

NoHumansRequired: Autonomous High-Quality Image Editing Triplet Mining — Kuprashevich et al. (2025) (arXiv:2507.14119, 2025)

What this evaluates

Evaluates the quality of AI-generated image edits by measuring semantic consistency, perceptual quality, and overall fidelity against source images and edit instructions.

Datasets

• GEdit-Bench — total ?; splits: test (-1)
• ImgEdit-Bench — total ?; splits: test (-1)

Metrics

• Semantic Consistency (SC) — range: [0, 10]
  • LLM-as-a-judge score (GPT-4o) measuring how well the edited image preserves the original semantic content relative to the source image and edit prompt.
• Perceptual Quality (PQ) — range: [0, 10]
  • LLM-as-a-judge score (GPT-4o) measuring the visual fidelity and aesthetic quality of the edited image.
• Overall (GEdit-Bench) (primary) — range: [0, 10]
  • LLM-as-a-judge score (GPT-4o) aggregating semantic consistency and perceptual quality into a single overall edit quality rating.
• Overall (ImgEdit-Bench) — range: [1, 5]
  • LLM-as-a-judge score (GPT-4o) averaging multiple edit quality criteria, each rated on a 1-to-5 scale.

Input / output format

Input: Source image and edit instruction/prompt (implied by image editing task; model generates an edited image).

Output: Edited image, evaluated by GPT-4o against source image and prompt to produce criterion scores.

Scoring recipe

def compute_metrics(predictions, gold, benchmark):
    scores = []
    for pred_img, src_img, prompt in zip(predictions, gold['src'], gold['prompt']):
        if benchmark == 'GEdit-Bench':
            sc = llm_judge(pred_img, src_img, prompt, 'Semantic Consistency', scale=10)
            pq = llm_judge(pred_img, src_img, prompt, 'Perceptual Quality', scale=10)
            ov = llm_judge(pred_img, src_img, prompt, 'Overall', scale=10)
            scores.append((sc, pq, ov))
        elif benchmark == 'ImgEdit-Bench':
            crit_scores = [llm_judge(pred_img, src_img, prompt, c, scale=5) for c in criteria]
            scores.append(mean(crit_scores))
    return mean(scores), std(scores), ci95(scores)

Common pitfalls

• LLM-as-a-judge scoring is sensitive to prompt wording and model version; must strictly use the official VIEScore and ImgEdit-Bench evaluation prompts.
• Benchmarks use different rating scales (0–10 vs. 1–5), so scores cannot be directly compared across datasets without normalization.
• Reported results average over 3 inference seeds; single-seed evaluations may not capture model variance or confidence intervals.

Evidence (verbatim from paper)

For GEdit-Bench, we use the VIEScore setup with GPT-4o to report Semantic Consistency (SC, 0-10), Perceptual Quality (PQ, 0-10), and Overall (O). For the ImgEdit-Bench evaluation, we adopt the original authors’ protocol: GPT-4o is used to score edited images across several criteria, each rated on a 1-to-5 scale.

Citation

@misc{kuprashevich2025nohumansrequired,
  title={NoHumansRequired: Autonomous High-Quality Image Editing Triplet Mining},
  author={Kuprashevich et al. (2025)},
  year={2025},
  note={arXiv:2507.14119}
}

• arXiv: 2507.14119