Midam Mil Auc Eval

midam-mil-auc-eval

Provable Multi-instance Deep AUC Maximization with Stochastic Pooling — Zhu et al. (2023) (arXiv:2305.08040, 2023)

What this evaluates

Evaluates multi-instance learning models for AUC maximization across tabular, histopathological, and medical image datasets. It probes the model's ability to handle large bags of instances via stochastic pooling while optimizing a min-max margin AUC loss.

Datasets

• MUSK1 — total 92; splits: train (-1), test (-1)
• MUSK2 — total 102; splits: train (-1), test (-1)
• Elephant — total 200; splits: train (-1), test (-1)
• Fox — total 200; splits: train (-1), test (-1)
• Tiger — total 200; splits: train (-1), test (-1)
• Breast Cancer — total 58; splits: train (-1), test (-1)
• Colon Ade. — total 1100; splits: train (-1), test (-1)
• PDGM — total 458; splits: train (-1), test (-1)
• OCT — total 2682; splits: train (-1), test (-1)

Metrics

• testing AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Computed as the fraction of positive-negative score pairs where the positive score is greater than the negative score.

Input / output format

Input: Bags of feature vectors or image patches, each bag associated with a single binary label (positive/negative).

Output: A single scalar prediction score per bag, passed through a sigmoid function.

Scoring recipe

def compute_auc(predictions, labels):
    pos_scores = [p for p, l in zip(predictions, labels) if l == 1]
    neg_scores = [p for p, l in zip(predictions, labels) if l == 0]
    auc = 0.0
    for ps in pos_scores:
        for ns in neg_scores:
            if ps > ns: auc += 1.0
            elif ps == ns: auc += 0.5
    return auc / (len(pos_scores) * len(neg_scores))

Common pitfalls

• Model selection uses the checkpoint with the largest validation AUC, not the final epoch.
• Deterministic pooling on high-resolution medical images causes GPU OOM; stochastic sampling is mandatory for baselines.
• Bag-level labels are used for evaluation, not instance-level ground truth.

Evidence (verbatim from paper)

We report the testing AUC based on a model with the largest validation AUC value. We uniformly randomly split the data with 0.9/0.1 train/test ratio and run 5-fold-cross-validation experiments with 3 different random seeds (totally 15 different trials).

Citation

@misc{zhu2023midam,
  title={Provable Multi-instance Deep AUC Maximization with Stochastic Pooling},
  author={Zhu et al. (2023)},
  year={2023},
  note={arXiv:2305.08040}
}

• arXiv: 2305.08040