Collaborative 3d Detection Eval

collaborative-3d-detection-eval

Towards Collaborative Autonomous Driving: Simulation Platform and End-to-End System — Liu et al. (2024) (arXiv:2404.09496, 2024)

What this evaluates

Evaluates collaborative 3D object detection performance and communication efficiency under various conditions including homogeneous/heterogeneous sensor setups, bandwidth constraints, communication latency, and pose errors.

Datasets

• DAIR-V2X — total ?; splits: test (-1)
• V2V4Real — total 20000; splits: test (-1)
• TUMTraf-V2X — total ?; splits: test (-1)
• OPV2V — total 12000; splits: test (-1)
• V2X-SIM2.0 — total 47200; splits: test (-1)

Metrics

• Average Precision (AP) at IoU 0.30/0.50 (primary) — range: [0, 1]
  • Standard object detection metric computing precision-recall curve area at specified IoU thresholds (0.30 and 0.50) between predicted and ground-truth 3D bounding boxes.
• Mean Average Precision (mAP) in BEV (primary) — range: [0, 1]
  • Average of AP scores across all object classes, computed in the Bird's Eye View (BEV) perspective using center distance for matching.
• Communication cost — range: bits
  • Calculated as log2(H × W × ||M||1 × C × 32 / 8) bits, where H and W are feature map dimensions, M is the selection matrix for transmitted features, and C is the number of channels.

Input / output format

Input: Multi-agent sensor data (LiDAR point clouds and/or RGB images) with corresponding poses, targeting a predefined spatial detection area.

Output: 3D bounding boxes with class labels and confidence scores for detected objects within the detection area.

Scoring recipe

def compute_ap_mAP(predictions, ground_truth, iou_thresh=0.5):
    # 1. Match predictions to ground truth boxes using IoU > iou_thresh
    # 2. Sort matches by confidence score descending
    # 3. Compute precision and recall at each threshold
    # 4. Interpolate precision-recall curve to get AP
    # 5. Average AP across all classes for mAP
    return ap, mAP

Common pitfalls

• Focusing solely on detection accuracy while ignoring the communication bandwidth trade-off.
• Evaluating only homogeneous sensor setups, neglecting heterogeneous configurations (e.g., LiDAR-only vs. camera-only agents).
• Assuming perfect synchronization and pose alignment, failing to test robustness against realistic communication latency and pose errors.

Evidence (verbatim from paper)

Detection performance. Following the collaborative perception methods [10], [11], [20], [44], the detection results are evaluated by 1) Average Precision (AP) at Intersection-over-Union (IoU) thresholds of 0.30, 0.50. 2) Mean average precision (mAP) in BEV perspective, considering the BEV center distance.

Citation

@misc{liu2024collaborative,
  title={Towards Collaborative Autonomous Driving: Simulation Platform and End-to-End System},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2404.09496}
}

• arXiv: 2404.09496