Ideabench Eval

ideabench-eval

IDEA-Bench: How Far are Generative Models from Professional Designing? — Chen Liang et al. (2024) (arXiv:2412.11767, 2024)

What this evaluates

Evaluates the professional design capabilities of generative models across text-to-image, image-to-image, and multi-image generation tasks. It probes aesthetic quality, contextual relevance, multimodal alignment, and adherence to complex, real-world design requirements that go beyond basic generation.

Datasets

• IDEA-Bench — total 275; splits: test (275); repo https://github.com/ali-vilab/IDEA-Bench

Metrics

• Avg. Score (primary) — range: [0, 100]
  • The average score across all subtasks and categories. Each subtask score is computed by averaging hierarchical evaluation question scores (0-100 scale), and the final metric is the mean of these subtask scores.

Input / output format

Input: Text prompts for generation tasks; input images plus text instructions for editing tasks; multiple input images and text for multi-image tasks.

Output: Generated image(s) corresponding to the provided prompt or instruction.

Scoring recipe

# Pseudo-code for IDEA-Bench scoring

total_score = 0
task_count = 0

for task in benchmark_tasks:
    subtask_scores = []
    for question in task.hierarchical_questions:
        # MLLM (e.g., GPT-4o) evaluates generated image against gold/reference
        score = mllm_evaluate(generated_image, question, gold_reference)
        subtask_scores.append(score)
    task_score = mean(subtask_scores)
    total_score += task_score
    task_count += 1

final_score = total_score / task_count
return final_score

Common pitfalls

• Relying on traditional metrics like FID or CLIPScore, which fail to capture professional design nuances and contextual relevance.
• Assuming basic T2I models can handle multi-image or complex editing tasks without MLLM-assisted prompt rephrasing.
• Overlooking the hierarchical evaluation structure (1,650 questions across 275 cases), which requires granular assessment rather than single-pass generation.

Evidence (verbatim from paper)

| Method | Subtasks Score | | | | | | | | | | | Avg. Score |

Each task category is averaged across all its subtasks, with the top-ranked model scores for each subtask highlighted in bold.

Citation

@misc{liang2024ideabench,
  title={IDEA-Bench: How Far are Generative Models from Professional Designing?},
  author={Chen Liang et al. (2024)},
  year={2024},
  note={arXiv:2412.11767}
}

• arXiv: 2412.11767