Cppe 5 Eval

cppe-5-eval

CPPE-5: Medical Personal Protective Equipment Dataset — Dagli et al. (2021) (arXiv:2112.09569, 2021)

What this evaluates

Evaluates object detection models on fine-grained medical personal protective equipment (PPE) in complex, real-world scenes. It probes a model's ability to localize and classify coveralls, face shields, gloves, masks, and goggles from non-canonical perspectives, measuring detection accuracy across multiple IoU thresholds and object scales.

Datasets

• CPPE-5 — total 1129; splits: train (1029), test (100)

Metrics

• AP (mean Average Precision) (primary) — range: [0, 100]
  • COCO-style mean Average Precision averaged over IoU thresholds from 0.50 to 0.95 in 0.05 increments. Computed per class and then macro-averaged across the five PPE categories.

Input / output format

Input: RGB images of complex, real-world scenes containing medical PPE instances, with ground-truth bounding boxes and class labels for coveralls, face shields, gloves, masks, and goggles.

Output: List of detected bounding boxes with class labels and confidence scores per image.

Scoring recipe

def compute_coco_ap(predictions, ground_truth):
    aps = []
    for iou_th in np.arange(0.50, 0.96, 0.05):
        tp, fp = 0, 0
        used_preds = set()
        for gt in ground_truth:
            best_idx = -1
            max_iou = 0
            for i, pred in enumerate(predictions):
                if i not in used_preds and pred['iou'] > max_iou:
                    max_iou = pred['iou']
                    best_idx = i
            if best_idx != -1 and max_iou >= iou_th:
                tp += 1
                used_preds.add(best_idx)
            else:
                fp += 1
        aps.append(tp / (tp + fp) if (tp + fp) > 0 else 0.0)
    return np.mean(aps) * 100

Common pitfalls

• Models are evaluated on non-canonical, real-world perspectives, so standard COCO-trained detectors often underperform without domain-specific fine-tuning.
• Scale-specific AP (S, M, L) requires correct object size binning during IoU matching, which varies by implementation and can cause score discrepancies.
• FPS is reported on a single V100 GPU, making cross-platform or cross-batch-size latency comparisons invalid.

Evidence (verbatim from paper)

For evaluation, we adopt the metrics from the COCO detection evaluation criteria, including the mean Average Precision (AP) across IoU thresholds ranging from 0.50 to 0.95 at different scales which are standard for object detection tasks. The inference speed FPS (Frames per second) for the detector is measured on a machine with 1 Tesla V100 GPU.

Citation

@misc{dagli2021cppe5,
  title={CPPE-5: Medical Personal Protective Equipment Dataset},
  author={Dagli et al. (2021)},
  year={2021},
  note={arXiv:2112.09569}
}

• arXiv: 2112.09569