# Benchmd Eval

> Evaluates modality-agnostic models across 19 real-world medical datasets spanning 1D, 2D, and 3D modalities. Probes performance under data scarcity (few-shot linear evaluation and finetuning) and out-of-distribution generalization across different hospitals and data distributions. Use when the user wants to benchmark on BenchMD, or asks about evaluating this task. Reports AUROC.

- Skill: `qhjqhj00/benchmd-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/benchmd-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/benchmd-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/benchmd-eval

---


# benchmd-eval

> BenchMD: A Benchmark for Unified Learning on Medical Images and Sensors — Wantlin et al. (2023) (arXiv:2304.08486, 2023)

## What this evaluates

Evaluates modality-agnostic models across 19 real-world medical datasets spanning 1D, 2D, and 3D modalities. Probes performance under data scarcity (few-shot linear evaluation and finetuning) and out-of-distribution generalization across different hospitals and data distributions.

## Datasets

- **BenchMD** — total 19; splits: train (-1), val (-1), test (-1); repo https://github.com/rajpurkarlab/BenchMD

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve. For multi-class tasks, computed as the unweighted average of per-class AUROC scores. For multi-label tasks, similarly averaged across classes.

## Input / output format

**Input**: Raw medical data (1D sensor signals, 2D images, or 3D volumes) processed through modality-specific embedding modules into a unified 256-dimensional space. Models are trained on labeled source data and evaluated zero-shot on unlabeled OOD target data.

**Output**: Class probabilities or logits from a linear classifier head or finetuned backbone, used to compute per-class AUROC scores.

## Scoring recipe

```python
def compute_auroc(y_true, y_pred, task_type='single'):
    scores = []
    classes = np.unique(y_true)
    for c in classes:
        y_bin = (y_true == c).astype(int)
        scores.append(roc_auc_score(y_bin, y_pred[:, c]))
    return np.mean(scores)
```

## Common pitfalls

- Performance is highly modality-dependent; no single SSL or pretraining method dominates across all 19 datasets.
- In-distribution validation performance does not reliably predict out-of-distribution test performance due to distribution shifts across hospitals.
- Label availability settings (8, 64, 256 per class) are applied differently for single-label vs multi-label tasks, which can cause overfitting or performance drops on OOD data.

## Evidence (verbatim from paper)

> We evaluate our models using AUROC score as the metric (taking an unweighted average of per-class scores for multi-class tasks).

## Citation

```bibtex
@misc{wantlin2023benchmd,
  title={BenchMD: A Benchmark for Unified Learning on Medical Images and Sensors},
  author={Wantlin et al. (2023)},
  year={2023},
  note={arXiv:2304.08486}
}
```

- arXiv: 2304.08486