Frechet Inception Distance

frechet-inception-distance

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium — Heusel et al. (2017) (arXiv:1706.08500, 2017)

What this evaluates

Measures the distance between the feature distributions of real and generated images using a pretrained Inception network. It evaluates both the fidelity and diversity of generated samples by comparing their mean and covariance in the feature space.

Datasets

• CelebA — total ?; splits: train (-1)
• CIFAR-10 — total ?; splits: train (-1)
• SVHN — total ?; splits: train (-1)
• LSUN Bedrooms — total ?; splits: train (-1)

Metrics

• FID (primary) — range: other
  • Computes the squared Fréchet distance between two multivariate Gaussians defined by the mean and covariance of Inception-v3 features extracted from real and generated images: d² = ||m - m_w||² + Tr(C + C_w - 2(C C_w)^{1/2}). Lower values indicate better quality.

Input / output format

Input: Real images from the dataset and 50,000 generated images from the trained GAN.

Output: A single scalar value representing the squared Fréchet distance between the two feature distributions.

Scoring recipe

real_features = inception_v3(real_images, layer='last_pooling')
gen_features = inception_v3(generated_images, layer='last_pooling')
m, C = real_features.mean(axis=0), np.cov(real_features, rowvar=False)
m_w, C_w = gen_features.mean(axis=0), np.cov(gen_features, rowvar=False)
diff = m - m_w
covmean = scipy.linalg.sqrtm(C.dot(C_w))
fid = diff.dot(diff) + np.trace(C + C_w - 2 * covmean)
return np.real(fid)

Common pitfalls

• Using the original Inception Score pooling layer instead of the last pooling layer specified in the paper.
• Stopping training at the final checkpoint rather than the checkpoint with the minimum FID, as FID can diverge or increase after optimal convergence.
• Using fewer than 50,000 generated samples, which leads to unstable covariance estimates and unreliable FID scores.

Evidence (verbatim from paper)

For computing the FID, we propagated all images from the training dataset through the pretrained Inception-v3 model following the computation of the Inception Score [53], however, we use the last pooling layer as coding layer. For this coding layer, we calculated the mean $m_w$ and the covariance matrix $C_w$. Thus, we approximate the first and second central moment of the function given by the Inception coding layer under the real world distribution. To approximate these moments for the model distribution, we generate 50,000 images, propagate them through the Inception-v3 model, and then compute the mean $m$ and the covariance matrix $C$.

Citation

@misc{heusel2017ttur,
  title={GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium},
  author={Heusel et al. (2017)},
  year={2017},
  note={arXiv:1706.08500}
}

• arXiv: 1706.08500