# Chexpert Atelectasis Eval

> Evaluates the diagnostic accuracy of predictive algorithms and human radiologists on chest X-ray images for detecting atelectasis. It specifically probes whether human expertise provides actionable, non-redundant information on input subsets where algorithms are algorithmically indistinguishable. Use when the user wants to benchmark on CheXpert, or asks about evaluating this task. Reports Matthews Correlation Coefficient (MCC).

- Skill: `qhjqhj00/chexpert-atelectasis-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/chexpert-atelectasis-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/chexpert-atelectasis-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/chexpert-atelectasis-eval

---


# chexpert-atelectasis-eval

> Human Expertise in Algorithmic Prediction — Alur et al. (2024) (arXiv:2402.00793, 2024)

## What this evaluates

Evaluates the diagnostic accuracy of predictive algorithms and human radiologists on chest X-ray images for detecting atelectasis. It specifically probes whether human expertise provides actionable, non-redundant information on input subsets where algorithms are algorithmically indistinguishable.

## Datasets

- **CheXpert** — total 224816; splits: train (224316), test (500); HF `stanfordmlgroup/chexpert`

## Metrics

- `Matthews Correlation Coefficient (MCC)` **(primary)** — range: [-1, 1]
  - Rescaled covariance between predicted and true binary labels. Computed as (TP*TN - FP*FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)).

## Input / output format

**Input**: Chest X-ray radiograph image

**Output**: Binary label: 1 (presence of atelectasis) or 0 (absence)

## Scoring recipe

```python
def compute_mcc(preds, gold):
    tp = sum(p == 1 and g == 1 for p, g in zip(preds, gold))
    tn = sum(p == 0 and g == 0 for p, g in zip(preds, gold))
    fp = sum(p == 1 and g == 0 for p, g in zip(preds, gold))
    fn = sum(p == 0 and g == 1 for p, g in zip(preds, gold))
    num = (tp * tn) - (fp * fn)
    den = math.sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn))
    return num / den if den > 0 else 0.0
```

## Common pitfalls

- Evaluating only average performance obscures the heterogeneity where humans significantly outperform algorithms on specific indistinguishable subsets.
- Bootstrap confidence intervals for algorithmic predictors within conditional subsets are not strictly valid because subsets are selected based on the predictions themselves.

## Evidence (verbatim from paper)

> Following Rajpurkar et al., (2021), we use the Matthew’s Correlation Coefficient (MCC) as a standard measure of binary classification accuracy (Chicco and Jurman, 2020). The MCC is simply the rescaled covariance between each prediction and the outcome, which corresponds to our definition of indistinguishability (Definition 3.1).

## Citation

```bibtex
@misc{alur2024humanexpertise,
  title={Human Expertise in Algorithmic Prediction},
  author={Alur et al. (2024)},
  year={2024},
  note={arXiv:2402.00793}
}
```

- arXiv: 2402.00793