Pascal Voc Detection Eval

pascal-voc-detection-eval

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks — Ren et al. (2015) (arXiv:1506.01497, 2015)

What this evaluates

Evaluates an object detection model's ability to localize and classify objects within images. It measures how well the system predicts bounding boxes and assigns correct class labels across multiple object categories.

Datasets

• PASCAL VOC 2007 — total 10000; splits: trainval (5000), test (5000)

Metrics

• mAP (primary) — range: percent
  • Mean Average Precision averaged over 20 object categories. For each class, Average Precision (AP) is computed as the area under the precision-recall curve at a fixed IoU threshold (typically 0.5 for VOC). mAP is the arithmetic mean of these AP values across all classes.

Input / output format

Input: Single RGB image

Output: List of predicted bounding boxes with associated class labels and confidence scores

Scoring recipe

def compute_mAP(predictions, ground_truth, iou_thresh=0.5):
    ap_scores = []
    for class_id in classes:
        preds_cls = sorted([p for p in predictions if p['class'] == class_id], key=lambda x: x['score'], reverse=True)
        gts_cls = [g for g in ground_truth if g['class'] == class_id]
        tp, fp, matched = 0, 0, set()
        for pred in preds_cls:
            best_iou, best_idx = 0, -1
            for i, gt in enumerate(gts_cls):
                if i not in matched:
                    iou = calculate_iou(pred['box'], gt['box'])
                    if iou > best_iou: best_iou, best_idx = iou, i
            if best_iou >= iou_thresh:
                tp += 1; matched.add(best_idx)
            else: fp += 1
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / len(gts_cls) if len(gts_cls) > 0 else 0
        ap_scores.append(trapezoidal_rule(prec, rec))
    return sum(ap_scores) / len(ap_scores) * 100

Common pitfalls

• Using proposal recall-to-IoU as the primary evaluation metric instead of final detection mAP, as the paper explicitly notes recall is only for diagnosing proposals.
• Failing to fix the number of proposals (e.g., 300 vs 2000) during testing, which significantly impacts mAP and breaks fair comparison.
• Not specifying whether the RPN and detector share convolutional features, as unshared variants yield lower mAP (~1-2% drop).

Evidence (verbatim from paper)

We primarily evaluate detection mean Average Precision (mAP), because this is the actual metric for object detection (rather than focusing on object proposal proxy metrics).

Citation

@misc{ren2015fasterrcnn,
  title={Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks},
  author={Ren et al. (2015)},
  year={2015},
  note={arXiv:1506.01497}
}

• arXiv: 1506.01497