Ice Bench Eval

ice-bench-eval

ICE-Bench: A Unified and Comprehensive Benchmark for Image Creating and Editing — Pan et al. (2025) (arXiv:2503.14482, 2025)

What this evaluates

Evaluates image generation and editing models across 31 fine-grained tasks spanning text-to-image creation, reference-guided creation, and various editing scenarios. It probes capabilities in aesthetic quality, imaging quality, prompt adherence, source/reference consistency, and controllability.

Datasets

• ICE-Bench — total ?; splits: (unstated)

Metrics

• prompt following (PF) (primary) — range: [0, 1]
  • Measures how well the generated image adheres to the text instruction. Higher values indicate better performance. Scores are computed per dimension and combined into a final task score via a weighted sum.
• aesthetic quality (AES) — range: [0, 1]
  • Assesses the visual appeal and artistic quality of the generated image. Higher values indicate better performance.
• imaging quality (IMG) — range: [0, 1]
  • Evaluates technical image quality, including clarity, resolution, and absence of artifacts. Higher values indicate better performance.
• reference consistency (REF) — range: [0, 1]
  • Measures how well the generated image preserves key features from a reference image (face, style, or subject). Higher values indicate better performance.
• source consistency (SRC) — range: [0, 1]
  • Assesses how well the generated image preserves the original content of the source image during editing. Higher values indicate better performance.
• controllability (CTRL) — range: [0, 1]
  • Evaluates the model's ability to follow specific structural or pose guidance during generation. Higher values indicate better performance.

Input / output format

Input: Text prompts, reference images (for reference-guided tasks), and source images with optional masks (for editing tasks).

Output: Generated images.

Scoring recipe

For each model-task pair:
  1. Identify applicable evaluation dimensions (AES, IMG, PF, REF, SRC, CTRL).
  2. If the model cannot perform the task, assign a score of 0.
  3. Otherwise, compute scores for each applicable dimension.
  4. Calculate final task score as a weighted sum of the dimension scores.
  5. Aggregate task scores for model-level comparison.

Common pitfalls

• Models are only evaluated on their designated tasks; unsupported tasks are assigned a score of zero, which can artificially lower averages if not explicitly handled.
• Default hyperparameters are used for all models to ensure consistency, which may not yield optimal performance for every architecture.
• Reference and source consistency metrics require careful alignment with specific task types (e.g., face vs. style vs. subject reference), and averaging across them can mask task-specific weaknesses.

Evidence (verbatim from paper)

We assess their performance from 3 aspects including aesthetic quality (AES), imaging quality (IMG) and prompt following (PF), as shown in Tab. 2. All evaluation metrics are designed such that higher values indicate better performance. For tasks that a model cannot address, its score is set to zero. The final score for each task is computed as a weighted sum of scores from multiple evaluation dimensions.

Citation

@misc{pan2025icebench,
  title={ICE-Bench: A Unified and Comprehensive Benchmark for Image Creating and Editing},
  author={Pan et al. (2025)},
  year={2025},
  note={arXiv:2503.14482}
}

• arXiv: 2503.14482