Conceptmix Eval

conceptmix-eval

ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty — Wu et al. (2024) (arXiv:2408.14339, 2024)

What this evaluates

Evaluates the compositional generalization capability of text-to-image models by testing their ability to generate images that satisfy multiple, simultaneously specified visual concepts (objects, colors, shapes, spatial relationships, etc.) within a single prompt. The benchmark probes model robustness to increasing compositional complexity (k) and reveals limitations in handling less frequent concept combinations.

Datasets

• ConceptMix — total 2100; splits: test (2100)

Metrics

• Full-mark score (primary) — range: [0, 1]
  • The proportion of generated images that correctly satisfy all k+1 required visual concepts specified in the prompt. Calculated as (number of fully correct images) / (total number of prompts).
• Concept fraction score — range: [0, 1]
  • The average proportion of visual concepts satisfied across all generated images for a given prompt set. Calculated as (sum of satisfied concepts per image) / (total concepts across all images).

Input / output format

Input: Text prompt containing at least one object and k additional visual concept categories (e.g., color, shape, spatial relationship, style, texture, size, number).

Output: Generated image corresponding to the input prompt.

Scoring recipe

def compute_full_mark_score(prompts, images, k_values):
    correct_count = 0
    for prompt, image, k in zip(prompts, images, k_values):
        # GPT-4o grades the image against the k+1 required concepts
        points_awarded = gpt4o_grade(image, prompt)  # returns int in [0, k+1]
        if points_awarded == k + 1:
            correct_count += 1
    return correct_count / len(prompts)

def compute_concept_fraction_score(prompts, images, k_values):
    total_satisfied = 0
    total_concepts = 0
    for prompt, image, k in zip(prompts, images, k_values):
        points_awarded = gpt4o_grade(image, prompt)
        total_satisfied += points_awarded
        total_concepts += (k + 1)
    return total_satisfied / total_concepts

Common pitfalls

• Pixel-level similarity metrics (e.g., CLIP score, FID) are inadequate for this benchmark because they fail to accurately capture spatial, shape, and size concepts, leading to misleading performance estimates.
• Evaluating models only at low complexity (k=1) masks the sharp performance degradation that occurs as compositional difficulty increases, particularly for open-source models.
• Model performance is heavily biased by concept frequency in training data (e.g., LAION-5B), causing uneven capabilities across concept categories that are not inherent to the model's architecture.

Evidence (verbatim from paper)

We consider two grading metrics: 1) Full-mark score, which measures the proportion of generated images where the image correctly satisfies all $k+1$ required concepts, and 2) Concept fraction score, which measures the average proportion of visual concepts satisfied by the generated images.

Citation

@misc{wu2024conceptmix,
  title={ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty},
  author={Wu et al. (2024)},
  year={2024},
  note={arXiv:2408.14339}
}

• arXiv: 2408.14339