Image Synthesis Eval

image-synthesis-eval

Scaling Rectified Flow Transformers for High-Resolution Image Synthesis — Esser et al. (2024) (arXiv:2403.03206, 2024)

What this evaluates

Evaluates the visual fidelity and text-image alignment of generated images. It measures realism and distribution matching using FID, and semantic alignment using CLIP scores.

Datasets

• COCO-2014 — total ?; splits: val (-1)

Metrics

• FID (CLIP features) (primary) — range: [0, ∞) (lower is better)
  • Fréchet Inception Distance computed on CLIP image features to measure the distance between real and generated image distributions. Lower values indicate better quality.
• CLIP score — range: [0, 1] (higher is better)
  • Cosine similarity between CLIP image and text embeddings. Higher values indicate better text-image alignment.

Input / output format

Input: Text prompt (for generation) and real images (for FID calculation).

Output: Generated RGB image.

Scoring recipe

def compute_metrics(gen_images, real_images, prompts):
    clip_gen = clip_model.encode(gen_images)
    clip_real = clip_model.encode(real_images)
    fid = frechet_distance(clip_gen, clip_real)
    clip_scores = cosine_similarity(clip_gen, clip_model.encode(prompts))
    return fid, clip_scores.mean()

Common pitfalls

• FID is calculated on CLIP features, not the standard Inception v3 features.
• Lower FID is better, while higher CLIP score is better.
• Evaluation is strictly performed on the COCO-2014 validation split, not the training data.

Evidence (verbatim from paper)

We calculate the FID on CLIP features as proposed by (Sauer et al., 2021). All metrics are evaluated on the COCO-2014 validation split (Lin et al., 2014).

Citation

@misc{esser2024scalingrectifiedflow,
  title={Scaling Rectified Flow Transformers for High-Resolution Image Synthesis},
  author={Esser et al. (2024)},
  year={2024},
  note={arXiv:2403.03206}
}

• arXiv: 2403.03206