Sku 110k Detection Eval

sku-110k-detection-eval

Precise Detection in Densely Packed Scenes — Goldman et al. (2019) (arXiv:1904.00853, 2019)

What this evaluates

Evaluates object detection and counting capabilities in densely packed scenes, specifically testing a model's ability to localize and count tightly overlapping items without false positives from standard non-maximum suppression.

Datasets

• SKU-110K — total ?; splits: train (-1), test (-1)
• CARPK — total ?; splits: test (-1)
• PUCPR+ — total ?; splits: test (-1)

Metrics

• AP (primary) — range: [0, 1]
  • Average Precision computed over predicted bounding boxes and ground truth annotations, typically averaged across IoU thresholds.
• AP.75 — range: [0, 1]
  • Average Precision computed strictly at an Intersection over Union (IoU) threshold of 0.75.
• AR300 — range: [0, 1]
  • Average Recall calculated with a maximum of 300 detections allowed per image.
• PR=.5 — range: [0, 1]
  • Precision measured at a recall threshold of 0.5 (or IoU=0.5 depending on convention).
• MAE — range: other
  • Mean Absolute Error between the predicted object count and the ground truth count per image.
• RMSE — range: other
  • Root Mean Square Error between the predicted object count and the ground truth count per image.

Input / output format

Input: RGB images of densely packed retail shelves or high-angle parking lot scenes.

Output: A list of bounding boxes with associated confidence or Soft-IoU scores, and optionally a single integer representing the total object count per image.

Scoring recipe

def compute_metrics(pred_boxes, pred_scores, gt_boxes, gt_counts):
    # Detection metrics via standard IoU matching and PR curve integration
    ap, ap75, ar300, pr5 = compute_detection_metrics(pred_boxes, pred_scores, gt_boxes)
    # Counting metrics
    mae = np.mean(np.abs(np.array(pred_counts) - np.array(gt_counts)))
    rmse = np.sqrt(np.mean((np.array(pred_counts) - np.array(gt_counts))**2))
    return {'AP': ap, 'AP.75': ap75, 'AR300': ar300, 'PR=.5': pr5, 'MAE': mae, 'RMSE': rmse}

Common pitfalls

• Standard NMS incorrectly suppresses valid detections for tightly packed/overlapping objects.
• Objectness confidence scores are less reliable than Soft-IoU scores for filtering false positives.
• Counting accuracy degrades significantly more than detection accuracy when using greedy NMS instead of overlap-aware merging.

Evidence (verbatim from paper)

Accuracy is reported using MAE and RMSE, as in our SKU-110K (Sec.[4]).

Citation

@misc{goldman2019precise,
  title={Precise Detection in Densely Packed Scenes},
  author={Goldman et al. (2019)},
  year={2019},
  note={arXiv:1904.00853}
}

• arXiv: 1904.00853