Openad Eval

openad-eval

OpenAD: Open-World Autonomous Driving Benchmark for 3D Object Detection — Zhongyu Xia et al. (arXiv:2411.17761, 2024)

What this evaluates

Evaluates open-world 3D object detection in autonomous driving by measuring detection accuracy and localization quality across seen and unseen object categories and domains.

Datasets

• OpenAD — total 2000; splits: test (2000); repo https://github.com/VDIGPKU/OpenAD

Metrics

• AP (primary) — range: [0, 1]
  • Mean Average Precision computed over IoU thresholds. It measures the precision of predicted bounding boxes across different recall levels.
• AR — range: [0, 1]
  • Average Recall at a fixed IoU threshold, measuring the fraction of ground truth objects successfully detected.
• ATE — range: other
  • Average Translation Error, the mean Euclidean distance between predicted and ground truth 3D box centers.
• ASE — range: other
  • Average Scale Error, the mean absolute difference between predicted and ground truth 3D box dimensions.

Input / output format

Input: Multi-view 2D images and/or LiDAR point clouds per scene, plus optional predefined category lists for open-vocabulary baselines.

Output: List of 3D bounding boxes with class labels, confidence scores, and translation/scale parameters.

Scoring recipe

def compute_3d_metrics(preds, gts, iou_thresh=0.5):
    matches = match_3d_boxes(preds, gts, iou_thresh)
    precisions = [tp / (tp + fp) for tp, fp in matches]
    recalls = [tp / (tp + fn) for tp, fn in matches]
    ap = interpolate_ap(precisions, recalls)
    ar = sum(tp for tp, fn in matches) / sum(tp + fn for tp, fn in matches)
    ate = mean(l2_dist(p.center, g.center) for p, g in matches)
    ase = mean(abs(p.scale - g.scale) for p, g in matches)
    return ap, ar, ate, ase

Common pitfalls

• Open-world models often predict non-driving objects (e.g., sky) or duplicate boxes for the same object, artificially lowering precision.
• Specialized models tend to overfit to in-domain benchmarks like nuScenes, failing to generalize to unseen categories or domains.
• Evaluating open-vocabulary methods requires correctly unioning common categories and adding generic queries like 'object that affects traffic' to avoid biased results.

Evidence (verbatim from paper)

Tables [2] and [3] show the evaluation results on 2D and 3D object detection models, including 2D and 3D open-world models, specialized models, and our baselines. The results show that current open-world models, irrespective of whether they are 2D or 3D detectors, tend to predict objects unrelated to driving (such as the sky) or to make repeated predictions for different parts of the same object, resulting in low precision and AP.

Citation

@misc{xia2024openad,
  title={OpenAD: Open-World Autonomous Driving Benchmark for 3D Object Detection},
  author={Zhongyu Xia et al.},
  year={2024},
  note={arXiv:2411.17761}
}

• arXiv: 2411.17761