Benchmd Eval

benchmd-eval

BenchMD: A Benchmark for Unified Learning on Medical Images and Sensors — Wantlin et al. (2023) (arXiv:2304.08486, 2023)

What this evaluates

Evaluates modality-agnostic models across 19 real-world medical datasets spanning 1D, 2D, and 3D modalities. Probes performance under data scarcity (few-shot linear evaluation and finetuning) and out-of-distribution generalization across different hospitals and data distributions.

Datasets

• BenchMD — total 19; splits: train (-1), val (-1), test (-1); repo https://github.com/rajpurkarlab/BenchMD

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve. For multi-class tasks, computed as the unweighted average of per-class AUROC scores. For multi-label tasks, similarly averaged across classes.

Input / output format

Input: Raw medical data (1D sensor signals, 2D images, or 3D volumes) processed through modality-specific embedding modules into a unified 256-dimensional space. Models are trained on labeled source data and evaluated zero-shot on unlabeled OOD target data.

Output: Class probabilities or logits from a linear classifier head or finetuned backbone, used to compute per-class AUROC scores.

Scoring recipe

def compute_auroc(y_true, y_pred, task_type='single'):
    scores = []
    classes = np.unique(y_true)
    for c in classes:
        y_bin = (y_true == c).astype(int)
        scores.append(roc_auc_score(y_bin, y_pred[:, c]))
    return np.mean(scores)

Common pitfalls

• Performance is highly modality-dependent; no single SSL or pretraining method dominates across all 19 datasets.
• In-distribution validation performance does not reliably predict out-of-distribution test performance due to distribution shifts across hospitals.
• Label availability settings (8, 64, 256 per class) are applied differently for single-label vs multi-label tasks, which can cause overfitting or performance drops on OOD data.

Evidence (verbatim from paper)

We evaluate our models using AUROC score as the metric (taking an unweighted average of per-class scores for multi-class tasks).

Citation

@misc{wantlin2023benchmd,
  title={BenchMD: A Benchmark for Unified Learning on Medical Images and Sensors},
  author={Wantlin et al. (2023)},
  year={2023},
  note={arXiv:2304.08486}
}

• arXiv: 2304.08486