Lnmbench Eval

lnmbench-eval

Benchmarking Real-World Medical Image Classification with Noisy Labels: Challenges, Practice, and Outlook — Ma et al. (2025) (arXiv:2512.09315, 2025)

What this evaluates

Evaluates the robustness of label noise learning (LNL) methods on medical image classification tasks. It probes model performance under varying noise types (symmetric, instance-dependent, real-world), noise ratios, and class imbalance distributions across multiple imaging modalities.

Datasets

• PathMNIST — total 107180; splits: train (89996), val (10004), test (7180)
• DermaMNIST — total 10015; splits: train (7008), val (1002), test (2005)
• BloodMNIST — total 17092; splits: train (11959), val (1712), test (3421)
• OrganCMNIST — total 23660; splits: train (13000), val (2392), test (8268)
• DRTiD — total 3100; splits: train (1600), val (400), test (1100)
• Kaggle DR+ — total 88702; splits: train (-1), val (-1), test (-1)
• CheXpert — total 224316; splits: train (-1), val (-1), test (-1)

Metrics

• average classification accuracy (primary) — range: percent
  • Computed as the mean classification accuracy over the last 5 training epochs for each method. Accuracy is calculated as the proportion of correctly predicted labels on the test set.

Input / output format

Input: RGB images resized to 224×224 (or 512×512 for DRTiD and Kaggle DR+) paired with noisy class labels.

Output: Predicted class label for each image.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

# Evaluated over the last 5 training epochs
epoch_accuracies = [compute_accuracy(preds_ep, test_labels) for ep in range(last_5_epochs)]
final_metric = sum(epoch_accuracies) / len(epoch_accuracies)

Common pitfalls

• Methods often overfit noisy labels in later training stages, causing a substantial discrepancy between validation and loss accuracies.
• Transition matrix estimation methods perform well on symmetric noise but collapse under instance-dependent noise due to their design assumptions.
• Real-world datasets use different annotation protocols (e.g., expert consensus vs. automated extraction), making direct cross-dataset comparison of noise levels difficult.

Evidence (verbatim from paper)

To obtain a stable and fair comparison, we report the average classification accuracy over the last 5 epochs for each method under three noise patterns: symmetric noise, instance-dependent noise, and real-world noise.

Citation

@misc{ma2025lnmbench,
  title={Benchmarking Real-World Medical Image Classification with Noisy Labels: Challenges, Practice, and Outlook},
  author={Ma et al. (2025)},
  year={2025},
  note={arXiv:2512.09315}
}

• arXiv: 2512.09315