Chexpert Atelectasis Eval

chexpert-atelectasis-eval

Human Expertise in Algorithmic Prediction — Alur et al. (2024) (arXiv:2402.00793, 2024)

What this evaluates

Evaluates the diagnostic accuracy of predictive algorithms and human radiologists on chest X-ray images for detecting atelectasis. It specifically probes whether human expertise provides actionable, non-redundant information on input subsets where algorithms are algorithmically indistinguishable.

Datasets

• CheXpert — total 224816; splits: train (224316), test (500); HF stanfordmlgroup/chexpert

Metrics

• Matthews Correlation Coefficient (MCC) (primary) — range: [-1, 1]
  • Rescaled covariance between predicted and true binary labels. Computed as (TPTN - FPFN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)).

Input / output format

Input: Chest X-ray radiograph image

Output: Binary label: 1 (presence of atelectasis) or 0 (absence)

Scoring recipe

def compute_mcc(preds, gold):
    tp = sum(p == 1 and g == 1 for p, g in zip(preds, gold))
    tn = sum(p == 0 and g == 0 for p, g in zip(preds, gold))
    fp = sum(p == 1 and g == 0 for p, g in zip(preds, gold))
    fn = sum(p == 0 and g == 1 for p, g in zip(preds, gold))
    num = (tp * tn) - (fp * fn)
    den = math.sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn))
    return num / den if den > 0 else 0.0

Common pitfalls

• Evaluating only average performance obscures the heterogeneity where humans significantly outperform algorithms on specific indistinguishable subsets.
• Bootstrap confidence intervals for algorithmic predictors within conditional subsets are not strictly valid because subsets are selected based on the predictions themselves.

Evidence (verbatim from paper)

Following Rajpurkar et al., (2021), we use the Matthew’s Correlation Coefficient (MCC) as a standard measure of binary classification accuracy (Chicco and Jurman, 2020). The MCC is simply the rescaled covariance between each prediction and the outcome, which corresponds to our definition of indistinguishability (Definition 3.1).

Citation

@misc{alur2024humanexpertise,
  title={Human Expertise in Algorithmic Prediction},
  author={Alur et al. (2024)},
  year={2024},
  note={arXiv:2402.00793}
}

• arXiv: 2402.00793