Object Detection Eval

object-detection-eval

Visual Semantic Information Pursuit: A Survey — Daqi Liu et al. (2019) (arXiv:1903.05434, 2019)

What this evaluates

Probes a model's ability to localize and classify objects within images by generating bounding boxes and assigning confidence scores. It evaluates both proposal quality and final detection accuracy across varying scales, occlusions, and natural contexts.

Datasets

• PASCAL VOC 2007 — total 4952; splits: test (4952)
• MS COCO — total 80000; splits: train (80000), val (5000)

Metrics

• mAP (primary) — range: [0, 1]
  • Mean Average Precision computed over all classes at a fixed IoU threshold (typically 0.5 for VOC 2007).
• AP — range: [0, 1]
  • Average Precision averaged over all categories and multiple IoU thresholds (0.5 to 0.95 step 0.05) for MS COCO.
• AP^50 — range: [0, 1]
  • Average Precision at a single IoU threshold of 0.5.
• AP^70 — range: [0, 1]
  • Average Precision at a single IoU threshold of 0.7.
• AR^1 — range: [0, 1]
  • Average Recall given 1 detection per image, computed per category.
• AR^10 — range: [0, 1]
  • Average Recall given 10 detections per image, computed per category.
• AR^100 — range: [0, 1]
  • Average Recall given 100 detections per image, computed per category.

Input / output format

Input: RGB image.

Output: List of predicted bounding boxes with class labels and confidence scores.

Scoring recipe

def compute_detection_metrics(preds, gts, iou_thresh):
    matches = []
    for gt in gts:
        best_pred = max(preds, key=lambda p: p.confidence if iou(p.box, gt.box) >= iou_thresh else 0)
        if best_pred and iou(best_pred.box, gt.box) >= iou_thresh:
            matches.append(True)
        else:
            matches.append(False)
    return sum(matches) / len(gts) if gts else 0

Common pitfalls

• mAP on VOC 2007 uses a fixed IoU=0.5, while COCO uses average over IoU 0.5:0.95; confusing the two leads to incomparable scores.
• AR metrics are computed per-category and then averaged, not globally over all detections.
• COCO validation splits vary (5k vs 10k images); results are not directly comparable across papers using different splits.

Evidence (verbatim from paper)

To evaluate the object detection methods, one needs to consider the following two performance measures: the object proposals generated by object detection methods and the corresponding objectness detection. In the existing literatures, the metrics for evaluating object proposals are often functions of intersection over union (IOU) between the proposal locations and the associated ground-truth annotations. Given the IOU, recall can be obtained as the fraction of ground-truth bounding boxes covered by proposal locations above a certain IOU overlap threshold. To evaluate the performance of the objectness detection, mean average precision (mAP) metric is often used for VOC 2007 test benchmark (the IOU threshold is normally set to 0.5), while MS COCO 2015 test-dev benchmark generally apply two types of metrics: average precision (AP) over all categories and different IOU thresholds, and average recall (AR) over all categories and IoUs...

Citation

@misc{liu2019visualsemantic,
  title={Visual Semantic Information Pursuit: A Survey},
  author={Daqi Liu et al. (2019)},
  year={2019},
  note={arXiv:1903.05434}
}

• arXiv: 1903.05434