Mmd Gan Eval

mmd-gan-eval

Demystifying MMD GANs — Binkowski et al. (2018) (arXiv:1801.01401, 2018)

What this evaluates

This protocol evaluates the sample quality and distributional alignment of Generative Adversarial Networks across standard image datasets. It probes the generator's ability to produce high-fidelity, diverse images that match real data distributions, using both classical feature-space metrics and a novel kernel-based distance measure.

Datasets

• MNIST — total ?; splits: test (10000)
• CIFAR-10 — total ?; splits: test (10000)
• LSUN — total ?; splits: test (25000)
• CelebA — total ?; splits: test (25000)

Metrics

• KID (primary) — range: [0, ∞)
  • Kernel Inception Distance: MMD^2 between generated and real image features using a polynomial or RBF kernel, averaged over 100 repetitions of sampling 1000 images without replacement.
• FID — range: [0, ∞)
  • Fréchet Inception Distance: Squared Fréchet distance between multivariate Gaussians fitted to Inception v3 features of real and generated images.
• Inception Score — range: [0, ∞)
  • exp(E_x[KL(p(y|x) || p(y))]): Measures image quality and diversity using the Inception v3 classifier's predicted label distribution.

Input / output format

Input: Real image dataset (MNIST, CIFAR-10, LSUN, or CelebA) and random noise vectors fed into the generator.

Output: Generated images matching the dataset resolution (28x28, 32x32, 64x64, or 160x160).

Scoring recipe

def compute_kid(real_imgs, gen_imgs, kernel='rbf', n_reps=100, n_samples=1000):
    kid_scores = []
    for _ in range(n_reps):
        r_sample = random.sample(real_imgs, n_samples)
        g_sample = random.sample(gen_imgs, n_samples)
        K_rr = kernel_matrix(r_sample)
        K_gg = kernel_matrix(g_sample)
        K_rg = kernel_matrix(r_sample, g_sample)
        mmd2 = K_rr.mean() + K_gg.mean() - 2 * K_rg.mean()
        kid_scores.append(mmd2)
    return np.mean(kid_scores)

Common pitfalls

• Inception Score is not meaningful for datasets like LSUN due to drastic domain mismatch with ImageNet class labels.
• Critic network size drastically impacts training speed and performance; MMD GANs achieve good results with small critics (16 filters) while WGAN-GP requires larger architectures.
• Gradient penalty scaling must be adjusted (1 vs 10) depending on the kernel to ensure fair comparison with baselines.

Evidence (verbatim from paper)

Quantitative scores are estimated based on 25000 generator samples (100000 for MNIST), and compared to 25000 dataset elements (for LSUN and CelebA) or the standard test set (10000 images held out from training for MNIST and CIFAR-10). Inception and FID scores were computed using 10 bootstrap resamplings of the given images; the KID score was estimated based on 100 repetitions of sampling 1000 elements without replacement.

Citation

@misc{binkowski2018demystifying,
  title={Demystifying MMD GANs},
  author={Binkowski et al. (2018)},
  year={2018},
  note={arXiv:1801.01401}
}

• arXiv: 1801.01401