Medmnist V2 Eval

medmnist-v2-eval

MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification — Yang et al. (2021) (arXiv:2110.14795, 2021)

What this evaluates

Evaluates the ability of machine learning models to classify biomedical images across diverse modalities, tasks, and scales. It probes generalization capabilities by testing on standardized 2D and 3D images resized to 28×28 or 28×28×28, covering binary, multi-class, multi-label, and ordinal regression tasks.

Datasets

• PathMNIST — total 107180; splits: train (89996), val (10004), test (7180); repo https://github.com/MedMNIST/MedMNIST
• ChestMNIST — total 112120; splits: train (78468), val (11219), test (22433); repo https://github.com/MedMNIST/MedMNIST
• DermaMNIST — total 10015; splits: train (7007), val (1003), test (2005); repo https://github.com/MedMNIST/MedMNIST
• OCTMNIST — total 109309; splits: train (97477), val (10832), test (1000); repo https://github.com/MedMNIST/MedMNIST
• PneumoniaMNIST — total 5856; splits: train (4708), val (524), test (624); repo https://github.com/MedMNIST/MedMNIST
• RetinaMNIST — total 1600; splits: train (1080), val (120), test (400); repo https://github.com/MedMNIST/MedMNIST
• BreastMNIST — total 780; splits: train (546), val (78), test (156); repo https://github.com/MedMNIST/MedMNIST
• BloodMNIST — total 17092; splits: train (11959), val (1712), test (3421); repo https://github.com/MedMNIST/MedMNIST
• TissueMNIST — total 236386; splits: train (165466), val (23640), test (47280); repo https://github.com/MedMNIST/MedMNIST
• OrganAMNIST — total 58850; splits: train (34581), val (6491), test (17778); repo https://github.com/MedMNIST/MedMNIST
• OrganCMNIST — total 23660; splits: train (13000), val (2392), test (8268); repo https://github.com/MedMNIST/MedMNIST
• OrganSMNIST — total 25221; splits: train (13940), val (2452), test (8829); repo https://github.com/MedMNIST/MedMNIST
• OrganMNIST3D — total 1743; splits: train (972), val (161), test (610); repo https://github.com/MedMNIST/MedMNIST
• NoduleMNIST3D — total 1633; splits: train (1158), val (165), test (310); repo https://github.com/MedMNIST/MedMNIST
• AdrenalMNIST3D — total 1584; splits: train (1188), val (98), test (298); repo https://github.com/MedMNIST/MedMNIST
• FractureMNIST3D — total 1370; splits: train (1027), val (103), test (240); repo https://github.com/MedMNIST/MedMNIST
• VesselMNIST3D — total 1909; splits: train (1335), val (192), test (382); repo https://github.com/MedMNIST/MedMNIST
• SynapseMNIST3D — total 1759; splits: train (1230), val (177), test (352); repo https://github.com/MedMNIST/MedMNIST

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly predicted labels out of the total number of test samples. For multi-label tasks, it is typically computed as subset accuracy or per-label accuracy depending on the convention used.

Input / output format

Input: 2D images of shape 28×28 (grayscale) or 28×28×3 (RGB), or 3D volumes of shape 28×28×28, provided as NumPy arrays.

Output: Single integer class label for binary/multi-class/ordinal tasks, or a binary vector of length L for multi-label tasks.

Scoring recipe

def compute_accuracy(predictions, labels):
    if labels.ndim == 2:  # multi-label
        correct = (predictions == labels).all(axis=1).sum()
    else:
        correct = (predictions == labels).sum()
    return correct / len(labels)

Common pitfalls

• Using image-level splits instead of patient-level splits where specified, causing data leakage.
• Ignoring the standardized 28×28/28×28×28 resizing, which significantly alters image content and model performance.
• Confusing multi-label classification with multi-class classification when computing accuracy.

Evidence (verbatim from paper)

Although it is fair to compare performance on the test set only, it could be expensive to compare the impact of the train-validation split. Therefore, we provide an official train-validation-test split for each subset. We use the official data split from source dataset (if provided) to avoid data leakage.

Citation

@misc{yang2021medmnistv2,
  title={MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification},
  author={Yang et al. (2021)},
  year={2021},
  note={arXiv:2110.14795}
}

• arXiv: 2110.14795