Chexpert Label Extraction Eval

chexpert-label-extraction-eval

CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison — Irvin et al. (2019) (arXiv:1901.07031, 2019)

What this evaluates

Evaluates an automated rule-based pipeline's ability to extract clinical observations from free-text radiology reports. It specifically tests the system's capacity to classify mentions as positive, negative, or uncertain, and to aggregate them into structured labels for 14 predefined observations.

Datasets

• CheXpert — total 224316; splits: evaluation (1000)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall for mention extraction, negation detection, and uncertainty classification. Macro- and micro-averages are computed across all 14 observations.

Input / output format

Input: Free-text radiology reports (specifically the Impression section)

Output: Structured labels for 14 observations: positive (1), negative, uncertain (u), or blank.

Scoring recipe

def compute_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Uncertainty and blank labels are frequently misclassified as negative or ignored in downstream tasks.
• The 1,000-report evaluation set is used only for labeler feasibility testing, not as a comprehensive benchmark.

Evidence (verbatim from paper)

We retrospectively collected chest radiographic studies from Stanford Hospital, performed between October 2002 and July 2017 in both inpatient and outpatient centers, along with their associated radiology reports. From these, we sampled a set of 1000 reports for manual review by a board-certified radiologist to determine feasibility for extraction of observations. Table 2: Performance of the labeler of NIH and our labeler on the report evaluation set on tasks of mention extraction, uncertainty detection, and negation detection, as measured by the F1 score.

Citation

@misc{irvin2019chexpert,
  title={CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison},
  author={Irvin et al. (2019)},
  year={2019},
  note={arXiv:1901.07031}
}

• arXiv: 1901.07031