Synthetic Medical Benchmark Eval

synthetic-medical-benchmark-eval

Overcoming Barriers to Data Sharing with Medical Image Generation: A Comprehensive Evaluation — August DuMont Schütte et al. (2020) (arXiv:2012.03769, 2020)

What this evaluates

Evaluates the quality and downstream utility of synthetic medical images generated by GANs by measuring how well classifiers trained on synthetic data perform compared to those trained on real data. It probes the trade-offs between image resolution, label complexity, and sample size on both visual fidelity and predictive performance.

Datasets

• Chest radiographs — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/AugustDS/synthetic-medical-benchmark
• Brain CT scans — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/AugustDS/synthetic-medical-benchmark

Metrics

• FID — range: other
  • Fréchet Inception Distance measuring distribution similarity between real and synthetic image feature embeddings. Computed batch-wise over a fixed N=10,000 images to avoid sample-size bias.
• mean AUC — range: [0, 1]
  • Mean area under the receiver operating characteristic curve averaged across all binary multi-label classes.
• AUC_real - AUC_syn (primary) — range: other
  • Difference in predictive performance between a classifier trained on real data and one trained on synthetic data, evaluated on a held-out real test set. Lower values indicate synthetic data yields performance closer to real data.

Input / output format

Input: Medical images (chest X-rays or brain CTs) with associated multi-label binary annotations. Classifiers receive images as input; GANs receive label vectors and noise.

Output: Predicted probabilities or binary labels for each class from the DenseNet-121 classifier. For GAN evaluation, generated synthetic images conditioned on input labels.

Scoring recipe

def compute_metric(real_test_preds, syn_test_preds, gold_labels):
    auc_real = compute_mean_auc(real_test_preds, gold_labels)
    auc_syn = compute_mean_auc(syn_test_preds, gold_labels)
    return auc_real - auc_syn

# Protocol:
# 1. Train DenseNet-121 on real train fold -> stop at val AUC convergence
# 2. Train DenseNet-121 on synthetic train fold -> stop at val AUC convergence
# 3. Evaluate both on held-out real test fold
# 4. Compute mean AUC over all labels for each
# 5. Return difference (AUC_real - AUC_syn)
# 6. Repeat across random initializations, report mean/std

Common pitfalls

• FID scores are highly sensitive to sample size; the protocol mandates fixing N=10,000 for both real and synthetic sets to avoid bias.
• Label overfitting in GANs at low sample counts (<3,000 per class) can artificially lower the AUC difference metric by making synthetic images too easy to classify but visually unrealistic.
• Higher resolutions (>128x128) improve realism but introduce artifacts that degrade classifier performance and privacy, requiring careful hyperparameter tuning.

Evidence (verbatim from paper)

In all settings, we used a pre-trained densenet-121 CNN as a predictive model, with the mean area under the receiver operating characteristics curve over all labels ($\overline{AUC}$) as the evaluation metric. For each classifier, we stopped training when the validation $\overline{AUC}$ converged. After the real predictive model is trained on the real dataset and the synthetic predictive models is trained on the synthetic dataset, we evaluated both on the separate, real data test fold to compute the difference in performance: $\overline{AUC}{\rm real}-\overline{AUC}{\rm syn}$.

Citation

@misc{dumontschutte2020overcoming,
  title={Overcoming Barriers to Data Sharing with Medical Image Generation: A Comprehensive Evaluation},
  author={August DuMont Schütte et al. (2020)},
  year={2020},
  note={arXiv:2012.03769}
}

• arXiv: 2012.03769