qhjqhj00 isic-ham-segmentation-eval
S-SYNTH: Knowledge-Based, Synthetic Generation of Skin Images — Kim et al. (2024) (arXiv:2408.00191, 2024)
What this evaluates
Evaluates dermatologic image segmentation models by measuring how training on real versus synthetic data affects performance on held-out real test sets, and how model accuracy correlates with controllable synthetic image parameters like skin tone and lesion shape.
Datasets
- ISIC — total ?; splits: train (-1), test (-1)
- HAM — total ?; splits: train (-1), test (-1)
Metrics
Dice score(primary) — range: [0, 1]- Measures overlap between predicted and ground-truth segmentation masks. Calculated as 2 * |A ∩ B| / (|A| + |B|), where A and B are the predicted and ground-truth masks respectively.
Input / output format
Input: Dermoscopic or clinical skin images (real or synthetic).
Output: Binary segmentation mask indicating the lesion region.
Scoring recipe
def dice_score(pred_mask, gt_mask):
intersection = np.logical_and(pred_mask, gt_mask).sum()
union = pred_mask.sum() + gt_mask.sum()
return 2.0 * intersection / union if union > 0 else 0.0
Common pitfalls
- Domain shift between real and synthetic images causes performance drops when training exclusively on synthetic data.
- Absolute performance on synthetic test images is lower than on real images, though relative trends remain comparable.
Evidence (verbatim from paper)
We systematically evaluated the effect of the training data composition on the real test set performance (similar to the popular Train-Synthetic-Test-Real (TSTR) protocol) on both ISIC and HAM. ... More importantly, each model that was trained on a particular subset of the ISIC (or HAM) and supplemented with synthetic images resulted in a higher Dice score than the model that was trained on the same subset of only the real images.
Citation
@misc{kim2024ssynth,
title={S-SYNTH: Knowledge-Based, Synthetic Generation of Skin Images},
author={Kim et al. (2024)},
year={2024},
note={arXiv:2408.00191}
}
- arXiv: 2408.00191