Mavrec Eval

mavrec-eval

Multiview Aerial Visual Recognition (MAVREC): Can Multi-view Improve Aerial Visual Perception? — Dutta et al. (2023) (arXiv:2312.04548, 2023)

What this evaluates

Evaluates object detection performance on aerial and ground-view imagery, probing how geographic context and multi-view data fusion affect detection accuracy across different object scales.

Datasets

• MAVREC — total 11024; splits: train (8605), val (805), test (1614)

Metrics

• mAP (primary) — range: [0, 100] percent
  • Mean Average Precision computed over object categories, averaging the area under the precision-recall curve across IoU thresholds. The paper references COCO-style evaluation [44], which typically averages AP over IoU thresholds from 0.50 to 0.95. Sub-metrics include AP at IoU=0.50 (AP_50), and AP for small (AP_S) and medium (AP_M) objects.

Input / output format

Input: Image frames from either ground or aerial perspectives, provided as labeled pairs for supervised training, labeled/unlabeled pairs for semi-supervised training, or single annotated frames for validation/testing.

Output: Predicted bounding boxes with class labels and confidence scores for each detected object in the input image.

Scoring recipe

def compute_mAP(predictions, ground_truths, iou_thresh=0.5):
    ap_scores = []
    for cls in classes:
        gt = ground_truths[cls]
        pred = predictions[cls]
        pred = sorted(pred, key=lambda x: x.confidence, reverse=True)
        tp, fp = [], []
        for p in pred:
            best_iou = max(iou(p.box, g.box) for g in gt)
            if best_iou >= iou_thresh and not g.used:
                tp.append(1); g.used = True
            else:
                fp.append(1)
        ap_scores.append(calculate_ap(tp, fp))
    return sum(ap_scores) / len(ap_scores) * 100

Common pitfalls

• Assuming pre-training on large ground-view datasets (like COCO) generalizes well to aerial views without geographic alignment; the paper shows strong domain shift across geographies.
• Overlooking scale-specific performance: aerial views contain significantly smaller objects, leading to poor AP_S scores that mask overall mAP performance.
• Treating semi-supervised results as purely supervised: the curriculum learning phase uses labeled ground/aerial images in a specific burn-in order, which critically affects pseudo-label quality.

Evidence (verbatim from paper)

We evaluate the models with the widely used metric for object detection, mean average precision (mAP) [[44]]; see a detailed discussion in §C.2.

Citation

@misc{dutta2023mavrec,
  title={Multiview Aerial Visual Recognition (MAVREC): Can Multi-view Improve Aerial Visual Perception?},
  author={Dutta et al. (2023)},
  year={2023},
  note={arXiv:2312.04548}
}

• arXiv: 2312.04548