Imagen Coco Drawbench Eval

imagen-coco-drawbench-eval

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding — Saharia et al. (2022) (arXiv:2205.11487, 2022)

What this evaluates

Evaluates text-to-image generation models on photorealism, image-text alignment, and compositional reasoning using standard dataset metrics and human preference studies.

Datasets

• MS-COCO — total ?; splits: val (-1)
• DrawBench — total ?; splits: test (-1)

Metrics

• FID-30K (primary) — range: other
  • Fréchet Inception Distance computed between the feature distributions of 30,000 generated images and 30,000 real images from the validation set.
• Human preference rate — range: percent
  • Percentage of pairwise comparisons where raters prefer the model-generated image over the reference image for photorealism or alignment.

Input / output format

Input: Text prompts (captions for COCO, structured prompts for DrawBench)

Output: RGB images (256x256 for COCO evaluation, varying resolutions for DrawBench)

Scoring recipe

def compute_fid(gen_imgs, ref_imgs):
    return frechet_inception_distance(gen_imgs[:30000], ref_imgs[:30000])

def compute_human_pref(gen_imgs, ref_imgs, criterion):
    wins = sum(1 for g, r in zip(gen_imgs, ref_imgs) if rater_prefers(g, r, criterion))
    return (wins / len(gen_imgs)) * 100

Common pitfalls

• FID is computed on a fixed subset of 30,000 samples, not the full validation set.
• Human evaluation filters out prompts containing people for the 'no people' subset, which significantly affects photorealism scores.
• Guidance weights differ between base (1.35) and super-resolution (8.0) models, affecting sample quality and alignment scores.

Evidence (verbatim from paper)

Table 1: MS-COCO 256x256 FID-30K. We use a guidance weight of 1.35 for our 64x64 model, and a guidance weight of 8.0 for our super-resolution model. ... Imagen achieves state of the art zero-shot FID on COCO at 7.27... Table 2 reports the human evaluation to test image quality and alignment on the COCO validation set. ... We report the percentage of time raters prefer Model A, Model B, or are indifferent for both image fidelity and image-text alignment.

Citation

@misc{saharia2022imagen,
  title={Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding},
  author={Saharia et al. (2022)},
  year={2022},
  note={arXiv:2205.11487}
}

• arXiv: 2205.11487