Medical Dataset Distillation Eval

medical-dataset-distillation-eval

Progressive trajectory matching for medical dataset distillation — Yu et al. (2024) (arXiv:2403.13469, 2024)

What this evaluates

Evaluates the effectiveness of dataset distillation methods for medical imaging by training classification models on synthetic datasets and measuring their accuracy on held-out real test sets. It probes how well distilled images preserve class-discriminative features across diverse medical modalities, resolutions, and class imbalances.

Datasets

• COVID19-CXR — total 21165; splits: test (-1)
• SKIN-HAM — total 10015; splits: test (-1)
• BREAST-ULS — total 780; splits: test (-1)
• PATHMNIST — total ?; splits: test (-1)
• OCTMNIST — total ?; splits: test (-1)
• ORGAN3D — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Standard classification accuracy: the proportion of correctly predicted labels on the real test set.

Input / output format

Input: Synthetic image dataset generated by a distillation algorithm, used as training data for a target classification network.

Output: Mean and standard deviation of classification accuracy over 10 random network initializations on the real test set.

Scoring recipe

def compute_metric(synthetic_data, real_test_data):
    accs = []
    for _ in range(10):
        model = create_random_model()
        train(model, synthetic_data, epochs=1000, optimizer='SGD')
        accs.append(evaluate(model, real_test_data))
    return mean(accs), std(accs)

Common pitfalls

• Using a single training run instead of 10 random initializations and reporting mean/std.
• Ignoring the IPC (images per class) setting, as performance varies significantly across IPC=2 and IPC=10.
• Omitting differentiable augmentations during the training of distilled images, which are required for fair comparison.

Evidence (verbatim from paper)

Finally, in the evaluation stage after the synthetic dataset is generated, we train 10 randomly initialized network on it each time for 1000 training epochs using SGD optimizer, and report the mean and standard deviation of their accuracy on the real test set.

Citation

@misc{yu2024progressive,
  title={Progressive trajectory matching for medical dataset distillation},
  author={Yu et al. (2024)},
  year={2024},
  note={arXiv:2403.13469}
}

• arXiv: 2403.13469