Synomaly Medical Anomaly Eval

synomaly-medical-anomaly-eval

Synomaly Noise and Multi-Stage Diffusion: A Novel Approach for Unsupervised Anomaly Detection in Medical Images — Bi et al. (2024) (arXiv:2411.04004, 2024)

What this evaluates

Evaluates unsupervised anomaly detection in medical imaging by training a generative model on healthy images and reconstructing anomalous inputs. It measures how well the model localizes and segments pathological regions by comparing the reconstruction residuals against ground-truth anomaly masks.

Datasets

• BraTS 2023 (Brain MRI) — total 5159; splits: train (3743), test_anomalous (1000), test_healthy (416); repo https://github.com/yuan-12138/Synomaly
• LiTS (Liver CT) — total 7467; splits: train (5820), test_anomalous (1000), test_healthy (647); repo https://github.com/yuan-12138/Synomaly
• Carotid US — total 8684; splits: train (7306), test_anomalous (545), test_healthy (833); repo https://github.com/yuan-12138/Synomaly

Metrics

• Dice (primary) — range: [0, 1]
  • Dice = 2 * |A ∩ B| / (|A| + |B|), where A and B are the predicted and ground-truth anomaly masks. Measures overlap between predicted and actual anomaly regions.
• Precision — range: [0, 1]
  • Precision = TP / (TP + FP). Proportion of predicted anomaly pixels that are correctly identified as anomalies.
• Recall — range: [0, 1]
  • Recall = TP / (TP + FN). Proportion of actual anomaly pixels that are correctly detected by the model.

Input / output format

Input: Anomalous medical image (MRI, CT, or ultrasound) provided as input to the generative model. Ground-truth anomaly segmentation masks are provided only for evaluation.

Output: A reconstructed 'healthy' image and a binary anomaly segmentation mask generated by thresholding the pixel-wise difference between the blurred input and blurred reconstructed image.

Scoring recipe

def compute_metrics(pred_mask, gt_mask):
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = pred_mask.sum() + gt_mask.sum()
    dice = 2 * intersection / union if union > 0 else 0.0
    precision = intersection / pred_mask.sum() if pred_mask.sum() > 0 else 0.0
    recall = intersection / gt_mask.sum() if gt_mask.sum() > 0 else 0.0
    return dice, precision, recall

Common pitfalls

• Inference hyperparameters (noise steps T, Gaussian kernel n, threshold Th) are optimized per method via grid search, meaning reported results reflect best-case tuning rather than fixed out-of-the-box settings.
• The anomaly mask is derived from the difference between Gaussian-blurred input and reconstructed images, which smooths residuals and may obscure fine-grained anomaly boundaries.
• The Carotid US dataset is in-house and not publicly released, limiting external reproducibility and cross-dataset generalization checks.

Evidence (verbatim from paper)

The evaluation metrics include the Dice, Precision, and Recall score, to ensure an objective and comprehensive comparison of different approaches.

Citation

@misc{bi2024synomaly,
  title={Synomaly Noise and Multi-Stage Diffusion: A Novel Approach for Unsupervised Anomaly Detection in Medical Images},
  author={Bi et al. (2024)},
  year={2024},
  note={arXiv:2411.04004}
}

• arXiv: 2411.04004