Commoncanvas Eval

commoncanvas-eval

CommonCanvas: An Open Diffusion Model Trained with Creative-Commons Images — Gokaslan et al. (2023) (arXiv:2310.16825, 2023)

What this evaluates

Evaluates the image quality and text-image alignment of a text-to-image diffusion model trained on Creative-Commons licensed data, benchmarking it against Stable Diffusion 2 using both automated distribution metrics and human pairwise preference.

Datasets

• MS COCO — total ?; splits: test (-1)
• PartiPrompts — total ?; splits: test (-1)

Metrics

• Frechet Inception Distance (FID) — range: other
  • Measures the distance between feature distributions of real and generated images using an Inception network. Lower values indicate higher quality.
• Kernel Inception Distance (KID) — range: other
  • Similar to FID but uses a polynomial kernel on Inception features to better capture distribution differences. Lower values indicate higher quality.
• CLIP-FID — range: other
  • FID computed in the CLIP embedding space instead of Inception. Lower values indicate higher quality.
• CLIP Score — range: [0, 1]
  • Average cosine similarity between image and text embeddings from a CLIP model. Higher values indicate better alignment.
• User preference rate (primary) — range: percent
  • Fraction of times human raters select the CommonCanvas-generated image over the SD2-generated image for the same prompt in a pairwise comparison.

Input / output format

Input: Text prompt (from PartiPrompts or COCO validation captions)

Output: Generated image (512x512 resolution)

Scoring recipe

def compute_metrics(generated_images, real_images, captions):
    fid = frechet_inception_distance(generated_images, real_images)
    kid = kernel_inception_distance(generated_images, real_images)
    clip_fid = frechet_inception_distance(generated_images, real_images, clip_space=True)
    clip_score = mean_cosine_similarity(clip.encode_images(generated_images), clip.encode_texts(captions))
    return fid, kid, clip_fid, clip_score

def compute_preference_rate(prompts, cc_images, sd_images):
    votes = 0
    total = 0
    for prompt, cc_img, sd_img in zip(prompts, cc_images, sd_images):
        choice = rater_selects(prompt, cc_img, sd_img)  # Randomized order
        if choice == 'commoncanvas':
            votes += 1
        total += 1
    return (votes / total) * 100

Common pitfalls

• CLIP-FID and CLIP Score are biased toward models trained on web-scraped data (like LAION) due to distribution shift in caption styles, potentially favoring SD2 over CommonCanvas.
• Automated metrics like FID/CLIP Score often fail to capture domain-specific weaknesses (e.g., faces, photography, paintings) that human raters easily identify.
• Preference rates are reported as the fraction of times the target model is preferred over SD2, not as an absolute quality score; a rate near 50% indicates parity, not high quality.

Evidence (verbatim from paper)

We measure performance with three automated image quality metrics on the MS COCO dataset: Frechet Inception Distance (FID), Kernal Inception Distance (KID), and CLIP-FID. Additionally, CLIP Score was evaluated to understand the alignment between captions and their respective images. ... In this experiment, human raters were shown a prompt (selected randomly from the PartiPrompts prompts set) along with two generated images in randomized order, one from the reference model (SD2-base) and the other from a CommonCanvas model. Users were asked which generated image they preferred. We report the fraction of the time users selected the image generated by the CommonCanvas model over the corresponding generation from SD2 as the user preference rate for that model.

Citation

@misc{gokaslan2023commoncanvas,
  title={CommonCanvas: An Open Diffusion Model Trained with Creative-Commons Images},
  author={Gokaslan et al. (2023)},
  year={2023},
  note={arXiv:2310.16825}
}

• arXiv: 2310.16825