Artbench 10 Eval

artbench-10-eval

The ArtBench Dataset: Benchmarking Generative Models with Artworks — Liao et al. (2022) (arXiv:2206.11404, 2022)

What this evaluates

Evaluates the quality and diversity of synthetic images generated by models across ten distinct artistic styles. It probes a model's ability to capture class-conditional and unconditional data distributions while measuring trade-offs between sample fidelity and variety.

Datasets

• ArtBench-10 — total 60000; splits: test (60000); repo https://github.com/liaopeiyuan/artbench

Metrics

• Fréchet Inception Distance (FID) (primary) — range: other
  • Measures the distance between ground-truth and generated image feature distributions. Compared to IS, FID leverages the dataset information and is considered to be more consistent with the noise level and human perception.
• Inception Score (IS) — range: other
  • Measures the ability of a generative model to capture the whole data distribution and producing high quality samples for each single class. However, IS does not take the whole data distribution into consideration and does not reflect the diversity of generated images.
• Precision — range: [0, 1]
  • Precision is the percentage of generated images that fall into the estimated manifold of real images. Precision measures the quality of generated images.
• Recall — range: [0, 1]
  • Recall is the percentage of real images that fall into the estimated manifold of generated images. Recall measures the diversity.
• Kernel Inception Distance (KID) — range: other
  • Measures the maximum mean discrepancy (MMD) on images with a kernel function. KID shares some nice properties of FID such as being capable of reflecting the artifacts of images. It can also compare skewness between distributions and is an unbiased estimator.

Input / output format

Input: Random noise vector (and class label for conditional synthesis).

Output: Synthetic image (32×32 or 256×256 pixels).

Scoring recipe

def compute_metrics(real_images, generated_images, feature_extractor):
    feats_real = feature_extractor(real_images)
    feats_gen = feature_extractor(generated_images)
    
    # FID
    mu_r, sigma_r = np.mean(feats_real, axis=0), np.cov(feats_real, rowvar=False)
    mu_g, sigma_g = np.mean(feats_gen, axis=0), np.cov(feats_gen, rowvar=False)
    fid = np.sum((mu_r - mu_g)**2) + np.trace(sigma_r + sigma_g - 2*np.sqrt(sigma_r @ sigma_g))
    
    # Precision & Recall (manifold estimation)
    precision = len([g in manifold(feats_real) for g in feats_gen]) / len(feats_gen)
    recall = len([r in manifold(feats_gen) for r in feats_real]) / len(feats_real)
    
    # IS & KID computed similarly via feature distributions
    return fid, precision, recall

Common pitfalls

• IS does not account for the overall data distribution and can be inflated by mode collapse without reflecting true diversity.
• FID and KID are highly sensitive to the choice of feature extractor and batch size, making cross-paper comparisons difficult.
• Precision and Recall measure different aspects (quality vs. diversity) and should be interpreted jointly rather than as a single score.

Evidence (verbatim from paper)

We adopt the commonly used evaluation metrics Inception Score (IS) [43], Fréchet Inception Distance (FID) [14], Improved Precision and Recall [27], and Kernel Inception Distance (KID) [2] for evaluation. Inception Score (IS) [43] measures the ability of a generative model to capture the whole data distribution and producing high quality samples for each single class. Fréchet Inception Distance (FID) [14] measures the distance between ground-truth and generated image feature distributions. Compared to IS, FID leverages the dataset information and is considered to be more consistent with the noise level and human perception.

Citation

@misc{liao2022artbench,
  title={The ArtBench Dataset: Benchmarking Generative Models with Artworks},
  author={Liao et al. (2022)},
  year={2022},
  note={arXiv:2206.11404}
}

• arXiv: 2206.11404