Hopkins Rfo Bench Eval

hopkins-rfo-bench-eval

Dataset and Benchmark for Enhancing Critical Retained Foreign Object Detection — Wang et al. (2025) (arXiv:2507.06937, 2025)

What this evaluates

Evaluates object detection models on chest X-rays for identifying critical retained foreign objects (RFOs) like sponges and needles. It probes both classification accuracy and precise localization of rare medical anomalies under data-scarce conditions.

Datasets

• Hopkins RFOs Bench — total 144; splits: train (-1), test (-1); repo https://github.com/YuliWanghust/RFO_Bench

Metrics

• ACC (primary) — range: [0, 1]
  • Accuracy = (True Positives + True Negatives) / Total Instances. Measures the proportion of correctly classified images at a fixed confidence threshold.
• FNR — range: [0, 1]
  • False Negative Rate = False Negatives / (False Negatives + True Positives). Measures the proportion of actual RFO cases missed by the model.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Quantifies the model's ability to discriminate between positive and negative classes across all classification thresholds.
• FROC — range: [0, 1]
  • Free-response Receiver Operating Characteristic. Plots the true positive fraction against the average number of false positives per image across varying detection thresholds to evaluate localization performance.

Input / output format

Input: Chest X-ray radiograph images.

Output: Bounding box coordinates and class labels (presence of critical RFO) for each detected object.

Scoring recipe

def compute_metrics(predictions, ground_truth, iou_thresh=0.5):
    tp, fp, fn = 0, 0, 0
    for pred in predictions:
        if match_to_gold(pred, ground_truth, iou=iou_thresh): tp += 1
        else: fp += 1
    fn = len(ground_truth) - tp
    acc = (tp + tn) / len(ground_truth)  # tn derived from image-level negatives
    fnr = fn / (fn + tp)
    auc = compute_roc_auc(tp, fp)
    froc = compute_froc_curve(tp, fp, num_images=len(ground_truth))
    return acc, fnr, auc, froc

Common pitfalls

• Using excessive synthetic augmentation (e.g., 4,000 images) can degrade performance due to overfitting or diminishing returns, with 2,000 images often yielding optimal results.
• DDPM-based synthetic data frequently reduces detection performance compared to physics-based generation, highlighting fidelity and generalizability limitations for rare clinical features.
• Evaluation is strictly confined to the held-out Hopkins RFOs Bench testing set, so results do not generalize to external hospital datasets or different RFO types.

Evidence (verbatim from paper)

For classification, we use ACC, FNR, and AUC metrics. Localization performance is evaluated using the FROC metric.

Citation

@misc{wang2025rfo_bench,
  title={Dataset and Benchmark for Enhancing Critical Retained Foreign Object Detection},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2507.06937}
}

• arXiv: 2507.06937