Monai Generative Eval

monai-generative-eval

Generative AI for Medical Imaging: extending the MONAI Framework — Pinaya et al. (2023) (arXiv:2307.15208, 2023)

What this evaluates

Evaluates the adaptability, modularity, and downstream application capabilities of generative models (LDMs, VQ-VAE, ControlNets) across diverse 2D and 3D medical imaging modalities. It tests the framework's ability to generate high-fidelity synthetic data, perform conditional generation, detect out-of-distribution samples, and execute image translation and super-resolution tasks.

Datasets

• MIMIC-CXR — total 96161; splits: train (-1)
• CSAW-M — total 9523; splits: train (-1)
• UK Biobank — total 41162; splits: train (-1)
• Retinal OCT — total 84483; splits: train (-1)
• Medical Decathlon — total ?; splits: train (-1)

Metrics

• FID (primary) — range: other
  • Fréchet Inception Distance: computes the Fréchet distance between multivariate Gaussians fitted to feature representations of real and synthetic images.
• MS-SSIM — range: [0, 1]
  • Multi-Scale Structural Similarity Index: measures perceptual similarity between images across multiple resolutions. Used here for both reconstruction quality and sample diversity.
• CLIP Score — range: other
  • Measures cosine similarity between image and text embeddings from a pre-trained CLIP model to evaluate text-image alignment.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve: evaluates the model's ability to distinguish in-distribution from out-of-distribution samples based on image likelihoods.
• PSNR — range: other
  • Peak Signal-to-Noise Ratio: measures the ratio between the maximum possible power of a signal and the power of corrupting noise, used for image translation and super-resolution fidelity.
• MAE — range: other
  • Mean Absolute Error: average absolute difference between predicted and ground truth pixel values.

Input / output format

Input: 2D or 3D medical images (e.g., chest X-rays, mammograms, brain MRIs, OCT scans) or paired image/text prompts for conditional generation.

Output: Synthetic images matching the input dimensions, or likelihood scores/AUC for OOD detection, or translated/super-resolved images.

Scoring recipe

def compute_fid(real_imgs, synth_imgs):
    feats_r = encoder(real_imgs)
    feats_s = encoder(synth_imgs)
    mu_r, sig_r = feats_r.mean(0), np.cov(feats_r, rowvar=False)
    mu_s, sig_s = feats_s.mean(0), np.cov(feats_s, rowvar=False)
    return np.sum((mu_r - mu_s)**2) + np.trace(sig_r + sig_s - 2*np.sqrt(sig_r @ sig_s))

def compute_ms_ssim(img1, img2):
    return multi_scale_structural_similarity(img1, img2)

Common pitfalls

• FID measures distribution similarity, not per-image fidelity; a low FID does not guarantee high visual quality for every generated sample.
• MS-SSIM for diversity is computed between pairs of synthetic images, making it sensitive to sampling temperature and classifier-free guidance weights.
• CLIP score evaluation requires sweeping guidance weights, as higher weights improve text alignment but degrade FID, creating a trade-off curve rather than a single optimal value.
• OOD detection AUC is reported as 1.0 across all classes, likely due to the simplicity of likelihood-based separation on this specific setup, which may not generalize to harder OOD tasks.

Evidence (verbatim from paper)

we quantify the performance of the autoencoder using the MS-SSIM between the input image and its reconstruction. We evaluate the quality of the samples generated by the diffusion model regarding sample fidelity by measuring the FID between the distribution of synthetic images and our test sets, and the sample diversity by measuring the MS-SSIM between pairs of synthetic images.

Citation

@misc{pinaya2023generative,
  title={Generative AI for Medical Imaging: extending the MONAI Framework},
  author={Pinaya et al. (2023)},
  year={2023},
  note={arXiv:2307.15208}
}

• arXiv: 2307.15208