Recaptioning Image Gen Eval

recaptioning-image-gen-eval

A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation — Segalis et al. (2023) (arXiv:2310.16656, 2023)

What this evaluates

Evaluates the semantic fidelity, object accuracy, and prompt adherence of text-to-image generation models by comparing automated metrics and human ratings on standard benchmarks.

Datasets

• MS-COCO validation set — total ?; splits: val (-1)
• DrawBench — total ?; splits: test (-1)

Metrics

• FID (primary) — range: other
  • Fréchet Inception Distance measuring the distance between feature distributions of real and generated images. Lower is better.
• Semantic Object Accuracy — range: percent
  • Percentage of correctly identified objects in generated images matching the prompt. Higher is better.
• Counting Alignment errors — range: other
  • Average absolute difference between the number of objects in the prompt and the generated image. Lower is better.
• Positional Alignment — range: percent
  • Score measuring the correctness of object locations relative to the prompt. Higher is better.
• Clip score — range: [0, 1]
  • Cosine similarity between image and text embeddings from CLIP, measuring overall prompt adherence. Higher is better.
• Human Success Rate (Successful Images) — range: percent
  • Percentage of generated images across all prompts and seeds that raters judge as fully following the prompt.
• Human Prompt Success Rate (Successful Prompts) — range: percent
  • Percentage of prompts with at least one successful image generation out of four seeds.

Input / output format

Input: Text prompts sampled from the MS-COCO validation set or the DrawBench dataset.

Output: Four generated images per prompt, each created using a different random seed.

Scoring recipe

# Automated metrics (using publicly available code from ref [27])
fid = compute_frechet_inception_distance(real_images, generated_images)
clip_score = compute_clip_similarity(prompts, generated_images)
soa = compute_semantic_object_accuracy(generated_images, prompts)
ca = compute_counting_alignment_errors(generated_images, prompts)
pa = compute_positional_alignment(generated_images, prompts)

# Human evaluation
successful_images = 0
successful_prompts = 0
for prompt in prompts:
    images = generate(prompt, seeds=4)
    raters = [r.select_fully_following_image(images) for r in raters]
    if any(raters): successful_prompts += 1
    successful_images += sum(raters)
human_success_rate = successful_images / (len(prompts) * 4)
human_prompt_success_rate = successful_prompts / len(prompts)

Common pitfalls

• Inception Score (IS) and O-IS are omitted because they yield higher scores than real images for Stable Diffusion and perform poorly on multi-object datasets like MS-COCO.
• Improvements are attributed to caption quality rather than training data contamination, as the Alttext baseline uses the exact same images and shows no improvement.

Evidence (verbatim from paper)

We evaluated the performance and semantic capabilities of the RECAP model using a battery of metrics suggested by [[27]] (using their publicly available code) on the MS-COCO validation dataset. To assess overall generation quality we use the standard FID metric and observe that images generated with the RECAP model have a significantly better score. In addition, we assess the semantic capabilities of our model: to check that the model generates faithfully the requested objects we use Semantic Object Accuracy, to check the number of generated objects we use Counting Alignment errors, to check that the locations of objects are correct we use Positional Alignment, and finally to check the overall adherence to the prompt we measure Clip score.

Citation

@misc{segalis2023recaptioning,
  title={A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation},
  author={Segalis et al. (2023)},
  year={2023},
  note={arXiv:2310.16656}
}

• arXiv: 2310.16656