Csaw M Eval

csaw-m-eval

CSAW-M: An Ordinal Classification Dataset for Benchmarking Mammographic Masking of Cancer — Sorkhei et al. (2021) (arXiv:2112.01330, 2021)

What this evaluates

Evaluates models on ordinal classification of mammographic masking potential (levels 1–8) and their clinical utility in predicting interval and large invasive cancers. It probes the model's ability to respect ordinal relationships in breast tissue obscuration and correlate these estimates with cancer outcomes.

Datasets

• CSAW-M — total ?; splits: 5-fold cross-validation (-1); repo https://github.com/yueliukth/CSAW-M

Metrics

• average mean absolute error (AMAE) (primary) — range: [0, 7]
  • Measures the average distance of predicted classes with respect to the true classes. Computed as the mean of absolute differences between predicted and true ordinal labels.
• Kendall’s τb — range: [-1, 1]
  • Measures the correlation of two rankings based on the number of concordant and discordant pairs, adjusted for ties. Ranges from -1 (perfect inverse correlation) to 1 (perfect correlation).
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall used to assess performance at identifying low-masking (levels 1–2) and high-masking (levels 7–8) mammograms.
• AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve for predicting interval cancers, large invasive cancers, and their composite endpoint.
• Odds Ratio — range: [0, ∞)
  • Ratio of the odds of an event occurring in a prediction quartile group relative to the first quartile (reference group). Used to evaluate monotonic increase in cancer risk across prediction groups.

Input / output format

Input: 632×512 mammography images.

Output: Discrete masking level prediction (1–8) or probability distribution over levels. For clinical tasks, a continuous score computed as the weighted average of level probabilities, or quartile group assignment.

Scoring recipe

import numpy as np
from scipy.stats import kendalltau

def compute_amae(y_true, y_pred):
    return np.mean(np.abs(np.array(y_pred) - np.array(y_true)))

def compute_kendall_tau(y_true, y_pred):
    return kendalltau(y_true, y_pred).correlation

Common pitfalls

• Treating ordinal masking levels as independent categories ignores the clinical reality that confusing level 1 with 8 is far worse than confusing 1 with 2.
• Converting discrete ordinal predictions to continuous scores for Task 2 requires a specific weighted-average probability formula; naive rounding or argmax breaks the clinical correlation analysis.
• Odds ratio calculation depends on quartile binning of predictions; incorrect binning or reference group selection invalidates the clinical endpoint comparison.

Evidence (verbatim from paper)

We consider two metrics widely used to evaluate ordinal classification, (1) average mean absolute error (AMAE) which measures the average distance of predicted classes w.r.t. the true classes and is robust to class imbalance [[30]], and (2) Kendall’s τb [[31]] which measures the correlation of two rankings based on the number of concordant and discordant pairs. Kendall’s τb ranges from -1 (perfect inverse correlation) to 1 (perfect correlation), and 0 indicates no correlation.

Citation

@misc{sorkhei2021csawm,
  title={CSAW-M: An Ordinal Classification Dataset for Benchmarking Mammographic Masking of Cancer},
  author={Sorkhei et al. (2021)},
  year={2021},
  note={arXiv:2112.01330}
}

• arXiv: 2112.01330