Benchread Eval

benchread-eval

BenchReAD: A systematic benchmark for retinal anomaly detection — Lian et al. (2025) (arXiv:2507.10492, 2025)

What this evaluates

Evaluates retinal anomaly detection models across fundus photography and OCT modalities, testing their ability to distinguish normal from abnormal images and generalize to unseen anomalies under varying supervision levels.

Datasets

• Fundus Benchmark — total ?; splits: train (39749), val (200), test (3027); repo https://github.com/DopamineLcy/BenchReAD
• OCT Benchmark — total ?; splits: train (79542), val (200), test (3606); repo https://github.com/DopamineLcy/BenchReAD

Metrics

• AUC-ROC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, computed by plotting the true positive rate against the false positive rate across all classification thresholds on the test set.

Input / output format

Input: Raw retinal images (fundus photography or OCT scans) provided as input to the anomaly detection model.

Output: Continuous anomaly scores for each image, which are thresholded to produce binary normal/abnormal predictions.

Scoring recipe

def compute_auc(predictions, labels):
    fpr, tpr, _ = roc_curve(labels, predictions)
    return auc(fpr, tpr)

Common pitfalls

• Models trained on both normal and abnormal samples tend to overfit to seen anomaly features, leading to poor generalization on unseen anomalies.
• Evaluation must separately report performance on seen versus unseen anomaly categories to properly assess generalization capabilities.
• Strict partitioning of training data into labeled and unlabeled subsets is required for fair comparison across supervision levels.

Evidence (verbatim from paper)

Figure 2: ROC curves of distinguishing normal samples against all abnormal ones on test sets. Corresponding AUCs (%) are marked alongside 95% confidence intervals.

Citation

@misc{lian2025benchread,
  title={BenchReAD: A systematic benchmark for retinal anomaly detection},
  author={Lian et al. (2025)},
  year={2025},
  note={arXiv:2507.10492}
}

• arXiv: 2507.10492