Coco Voc Detection Eval

coco-voc-detection-eval

PhiNets: a scalable backbone for low-power AI at the edge — Paissan et al. (2021) (arXiv:2110.00337, 2021)

What this evaluates

Evaluates object detection performance of scalable neural backbones on resource-constrained edge devices by measuring mean Average Precision across varying computational budgets and low input resolutions.

Datasets

• MS COCO — total ?; splits: train (-1)
• VOC2012 — total ?; splits: train (-1)

Metrics

• mAP (primary) — range: percent
  • Mean Average Precision computed over predicted and ground truth bounding boxes. Standard COCO/VOC protocol averaged across IoU thresholds.

Input / output format

Input: Images resized to specific low resolutions (e.g., 128x128, 96x96, 160x160) containing only the 'person' class and objects with bounding box area > 1/64 of the original image.

Output: Predicted bounding boxes and confidence scores for each frame.

Scoring recipe

def compute_mAP(preds, gts):
    ap_scores = []
    for gt in gts:
        pr_curve = compute_precision_recall(preds, gt)
        ap_scores.append(trapezoidal_rule(pr_curve))
    return mean(ap_scores)

Common pitfalls

• Dataset is heavily filtered to only the 'person' class and objects larger than 1/64 of the image area, drastically reducing ground truth boxes compared to standard benchmarks.
• Input resolutions are constrained to very low values (96x96 to 160x160) for MCU inference, which is non-standard for general object detection evaluation and skews mAP expectations.

Evidence (verbatim from paper)

The platform is capable of running object detection at over 50 fps with the proposed hardware, at a power consumption from 1.3mW/fps (for networks achieving 53.7 / 60.3 mAP on the selected subsets of COCO/VOC datasets) to 11.8mW/fps (64.1 / 73.9 mAP on COCO/VOC), or, in other words, 10fps tracking at 13mW to 118mW, depending on the performance required by the specific application.

Citation

@misc{paissan2021phinet,
  title={PhiNets: a scalable backbone for low-power AI at the edge},
  author={Paissan et al. (2021)},
  year={2021},
  note={arXiv:2110.00337}
}

• arXiv: 2110.00337