Object Detection Synthetic Real Eval

object-detection-synthetic-real-eval

Improving Object Detector Training on Synthetic Data by Starting With a Strong Baseline Methodology — Ruis et al. (2024) (arXiv:2405.19822, 2024)

What this evaluates

Evaluates object detection models trained on synthetic data against real-world baselines, probing their ability to generalize across domains without explicit domain adaptation. It tests how architectural choices (Transformers vs CNNs) and data augmentation strategies impact detection accuracy on geometric versus texture-heavy features.

Datasets

• DGTA-VisDrone — total ?; splits: train (-1), test (-1)
• RarePlanes — total ?; splits: train (-1), test (-1)
• Vehicle Detection — total ?; splits: train (-1), test (-1)

Metrics

• mAP@50 (primary) — range: [0, 1]
  • Mean Average Precision computed at an Intersection over Union (IoU) threshold of 0.5. It averages the precision across all recall levels for each class.
• mAP — range: [0, 1]
  • Mean Average Precision averaged over multiple IoU thresholds (typically 0.50:0.95 in standard protocols, though dataset-specific).

Input / output format

Input: RGB images containing objects of interest (drones, airplanes, vehicles) with varying backgrounds, scales, and occlusions.

Output: Bounding box coordinates (x, y, width, height) and class labels for each detected object, along with confidence scores.

Scoring recipe

def compute_map(preds, gts, iou_thresh=0.5):
    aps = []
    for cls in classes:
        gt_cls = [g for g in gts if g['class'] == cls]
        tp, fp = 0, 0
        for p in sorted([x for x in preds if x['class'] == cls], key=lambda x: x['score'], reverse=True):
            if max(iou(p['box'], g['box']) for g in gt_cls) >= iou_thresh:
                tp += 1
            else:
                fp += 1
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / len(gt_cls) if gt_cls else 0
        aps.append(interpolate_ap(prec, rec))
    return sum(aps) / len(aps)

Common pitfalls

• Applying aggressive augmentations like large scale jittering or MixUp to datasets with fixed resolution and no occlusions (e.g., RarePlanes) can degrade detection accuracy.
• Reporting only mAP for vehicle detection datasets can be misleading, as partial detections (e.g., antennas, cabins) lower mAP but may still yield high mAP@50 scores.
• Comparing models without matching parameter counts or using inappropriate backbones (e.g., Swin-B without pretrained Faster-RCNN variants) leads to unfair baseline comparisons.

Evidence (verbatim from paper)

We report our results using both mAP and mAP@50. Due to the nature of our dataset, reporting solely the mAP would provide an inaccurate representation of the results, because a low mAP but high mAP@50 is observed to be the result of the exclusion of specific parts sticking out of the vehicle, such as an antenna, camera, or sometimes even the entire truck cabin.

Citation

@misc{ruis2024improving,
  title={Improving Object Detector Training on Synthetic Data by Starting With a Strong Baseline Methodology},
  author={Ruis et al. (2024)},
  year={2024},
  note={arXiv:2405.19822}
}

• arXiv: 2405.19822