Chexradinet Eval

chexradinet-eval

Using Radiomics as Prior Knowledge for Thorax Disease Classification and Localization in Chest X-rays — Yan Han et al. (arXiv:2011.12506, 2020)

What this evaluates

Evaluates a multi-task deep learning framework for thorax disease classification and weakly-supervised localization on chest X-rays. It probes the model's ability to detect multiple pathologies and accurately localize abnormal regions without requiring pre-annotated bounding boxes during training.

Datasets

• NIH Chest X-ray — total 112120; splits: train (-1), val (-1), test (-1)
• CheXpert — total 224316; splits: train (-1), val (-1), test (-1)
• MIMIC-CXR — total 377110; splits: train (-1), val (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic (ROC) curve. Computed per disease class and averaged across the 14 thoracic diseases for the mean score.
• T(IoU) — range: [0, 1]
  • Thresholded Intersection over Union accuracy. For a given IoU threshold, a prediction is counted as correct if the IoU between the predicted disease region mask and the ground truth bounding box meets or exceeds the threshold. Accuracy is the mean correct rate across all evaluated images.

Input / output format

Input: Chest X-ray images.

Output: Per image: multi-label disease classification probabilities and predicted disease region masks/activation maps for localization.

Scoring recipe

def compute_auc(y_true, y_pred_prob):
    return roc_auc_score(y_true, y_pred_prob)

def compute_t_iou_accuracy(ground_truth_boxes, predicted_masks, threshold):
    correct = 0
    for gt_box in ground_truth_boxes:
        pred_mask = get_predicted_mask(gt_box.image_id)
        iou = intersection_over_union(pred_mask, gt_box)
        if iou >= threshold:
            correct += 1
    return correct / len(ground_truth_boxes)

Common pitfalls

• Uncertain labels (-1) in CheXpert and MIMIC-CXR are explicitly removed before evaluation, not treated as negative samples.
• Localization is only evaluated on 8 diseases because bounding box annotations are only available for those 8 in the NIH dataset.
• Splits must be strictly patient-level to prevent data leakage; the paper explicitly states 'no patient overlap between the sets'.
• IoU threshold significantly impacts reported accuracy; clinical preference is for high thresholds (e.g., 0.7), where performance drops sharply compared to lenient thresholds like 0.1.

Evidence (verbatim from paper)

For the abnormality detection task, we randomly split each dataset into training (70%), validation (10%), and test (20%) sets. Note that there is no patient overlap between the sets. We use AUC scores, the area under the ROC curve, to measure the disease identification accuracy. A higher AUC score indicates better performance. For the abnormality localization task, following the work of Li et al., we only consider 8 diseases for the evaluation of mask generation because only eight types of diseases are provided with bounding boxes in the NIH Chest X-ray dataset. We use intersection over union (IoU) to evaluate the predicted disease regions against the ground truth bounding boxes.

Citation

@misc{han2020chexradinet,
  title={Using Radiomics as Prior Knowledge for Thorax Disease Classification and Localization in Chest X-rays},
  author={Yan Han et al.},
  year={2020},
  note={arXiv:2011.12506}
}

• arXiv: 2011.12506