Gamma Glaucoma Eval

gamma-glaucoma-eval

GAMMA Challenge:Glaucoma grAding from Multi-Modality imAges — Wu et al. (2022) (arXiv:2202.06511, 2022)

What this evaluates

Evaluates multi-modal medical image analysis models for glaucoma staging by jointly processing 2D fundus images and 3D OCT volumes. It probes the model's ability to fuse cross-modality features and correctly classify patients into normal, early, or progressive glaucoma stages.

Datasets

• GAMMA Challenge — total ?; splits: preliminary (-1), final test (-1)

Metrics

• kappa (primary) — range: percent
  • Cohen's or Weighted Kappa coefficient, expressed as a percentage. It measures agreement between model predictions and ground truth labels while correcting for chance. For this ordinal classification task, it is typically computed as weighted kappa to account for the clinical severity ordering of normal, early, and progressive glaucoma.

Input / output format

Input: Paired 2D fundus images and 3D OCT volumes for each patient.

Output: Predicted glaucoma stage: normal, early-glaucoma, or progressive-glaucoma.

Scoring recipe

def compute_kappa(preds, gold):
    labels = sorted(set(preds) | set(gold))
    conf = confusion_matrix(gold, preds, labels=labels)
    observed = np.trace(conf) / conf.sum()
    expected = np.dot(conf.sum(axis=0), conf.sum(axis=1)) / conf.sum()**2
    kappa = (observed - expected) / (1 - expected)
    return kappa * 100

Common pitfalls

• Preliminary set scores often overestimate final test performance due to overfitting; teams should be evaluated strictly on the held-out final test set.
• Overall accuracy can mask clinically critical errors; predicting progressive glaucoma as early is more dangerous than the reverse, so progressive-class accuracy should be prioritized in clinical deployment.
• The final competition ranking uses a weighted sum across three tasks (0.4glaucoma + 0.3fundus + 0.3*OCT), but the primary grading metric remains kappa.

Evidence (verbatim from paper)

The quantitative scores of the glaucoma grading task measured by kappa are shown in Table 3. We reported their performances in the preliminary stage (evaluation on preliminary set) and the final stage (evaluation on final test set).

Citation

@misc{wu2022gamma,
  title={GAMMA Challenge:Glaucoma grAding from Multi-Modality imAges},
  author={Wu et al. (2022)},
  year={2022},
  note={arXiv:2202.06511}
}

• arXiv: 2202.06511