# Csaw M Eval

> Evaluates models on ordinal classification of mammographic masking potential (levels 1–8) and their clinical utility in predicting interval and large invasive cancers. It probes the model's ability to respect ordinal relationships in breast tissue obscuration and correlate these estimates with cancer outcomes. Use when the user wants to benchmark on CSAW-M, or asks about evaluating this task. Reports average mean absolute error (AMAE).

- Skill: `qhjqhj00/csaw-m-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/csaw-m-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/csaw-m-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/csaw-m-eval

---


# csaw-m-eval

> CSAW-M: An Ordinal Classification Dataset for Benchmarking Mammographic Masking of Cancer — Sorkhei et al. (2021) (arXiv:2112.01330, 2021)

## What this evaluates

Evaluates models on ordinal classification of mammographic masking potential (levels 1–8) and their clinical utility in predicting interval and large invasive cancers. It probes the model's ability to respect ordinal relationships in breast tissue obscuration and correlate these estimates with cancer outcomes.

## Datasets

- **CSAW-M** — total ?; splits: 5-fold cross-validation (-1); repo https://github.com/yueliukth/CSAW-M

## Metrics

- `average mean absolute error (AMAE)` **(primary)** — range: [0, 7]
  - Measures the average distance of predicted classes with respect to the true classes. Computed as the mean of absolute differences between predicted and true ordinal labels.
- `Kendall’s τb` — range: [-1, 1]
  - Measures the correlation of two rankings based on the number of concordant and discordant pairs, adjusted for ties. Ranges from -1 (perfect inverse correlation) to 1 (perfect correlation).
- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall used to assess performance at identifying low-masking (levels 1–2) and high-masking (levels 7–8) mammograms.
- `AUC` — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve for predicting interval cancers, large invasive cancers, and their composite endpoint.
- `Odds Ratio` — range: [0, ∞)
  - Ratio of the odds of an event occurring in a prediction quartile group relative to the first quartile (reference group). Used to evaluate monotonic increase in cancer risk across prediction groups.

## Input / output format

**Input**: 632×512 mammography images.

**Output**: Discrete masking level prediction (1–8) or probability distribution over levels. For clinical tasks, a continuous score computed as the weighted average of level probabilities, or quartile group assignment.

## Scoring recipe

```python
import numpy as np
from scipy.stats import kendalltau

def compute_amae(y_true, y_pred):
    return np.mean(np.abs(np.array(y_pred) - np.array(y_true)))

def compute_kendall_tau(y_true, y_pred):
    return kendalltau(y_true, y_pred).correlation
```

## Common pitfalls

- Treating ordinal masking levels as independent categories ignores the clinical reality that confusing level 1 with 8 is far worse than confusing 1 with 2.
- Converting discrete ordinal predictions to continuous scores for Task 2 requires a specific weighted-average probability formula; naive rounding or argmax breaks the clinical correlation analysis.
- Odds ratio calculation depends on quartile binning of predictions; incorrect binning or reference group selection invalidates the clinical endpoint comparison.

## Evidence (verbatim from paper)

> We consider two metrics widely used to evaluate ordinal classification, (1) average mean absolute error (AMAE) which measures the average distance of predicted classes w.r.t. the true classes and is robust to class imbalance [[30]], and (2) Kendall’s τb [[31]] which measures the correlation of two rankings based on the number of concordant and discordant pairs. Kendall’s τb ranges from -1 (perfect inverse correlation) to 1 (perfect correlation), and 0 indicates no correlation.

## Citation

```bibtex
@misc{sorkhei2021csawm,
  title={CSAW-M: An Ordinal Classification Dataset for Benchmarking Mammographic Masking of Cancer},
  author={Sorkhei et al. (2021)},
  year={2021},
  note={arXiv:2112.01330}
}
```

- arXiv: 2112.01330