Nuscenes Hdmap Eval

nuscenes-hdmap-eval

End-to-End Vectorized HD-map Construction with Piecewise Bezier Curve — Limeng Qiao et al. (2023) (arXiv:2306.09700, 2023)

What this evaluates

This benchmark evaluates the accuracy of end-to-end vectorized high-definition map construction from multi-camera images. It probes a model's ability to precisely predict instance-level road elements (lane dividers, pedestrian crossings, road boundaries) as continuous curves rather than rasterized masks or polylines.

Datasets

• NuScenes — total 28130; splits: train (700), val (150)

Metrics

• mAP (primary) — range: percent
  • Instance-level Average Precision (AP) computed using Chamfer Distance between ground-truth and predicted curve instances. A prediction is a true positive if the Chamfer Distance is below a specified threshold. The overall mAP is the average of AP scores across three thresholds ([0.2, 0.5, 1.0] m) and three map categories (lane-divider, ped-crossing, road-boundary).

Input / output format

Input: Six surrounding camera images (360° FOV) per driving scene, resized to 896×512 pixels, covering a perception range of [30, 30, 15, 15] meters relative to the ego-vehicle.

Output: A sparse set of vectorized map elements, each represented as a piecewise Bézier curve defined by control point coordinates and a segmentation class (lane-divider, ped-crossing, or road-boundary).

Scoring recipe

def compute_mAP(predictions, ground_truth, thresholds=[0.2, 0.5, 1.0]):
    categories = ['lane-divider', 'ped-crossing', 'road-boundary']
    ap_scores = []
    for cat in categories:
        gt = [g for g in ground_truth if g.category == cat]
        preds = [p for p in predictions if p.category == cat]
        cat_ap = 0.0
        for thresh in thresholds:
            tp = sum(1 for p in preds if min(chamfer_distance(p, g) for g in gt) < thresh)
            cat_ap += tp / max(len(gt), 1)
        ap_scores.append(cat_ap / len(thresholds))
    return sum(ap_scores) / len(ap_scores) * 100

Common pitfalls

• The evaluation uses Chamfer Distance rather than point-wise IoU or Hausdorff distance, making results sensitive to curve parameterization and control point density.
• The perception range is strictly fixed to [30, 30, 15, 15]m with a resolution of 0.15 m/pixel; deviating from these bounds or resolutions breaks comparability.
• A secondary 'simpler' protocol uses thresholds [0.5, 1.0, 1.5]m, which significantly inflates AP scores compared to the standard [0.2, 0.5, 1.0]m protocol and should not be mixed in reporting.

Evidence (verbatim from paper)

We utilize the exact same evaluation protocol as [[24]] of average precision (AP) to access the map construction quality over the instance-level. To be concrete, given a pair of instances from ground-truth and predictions respectively, this protocol computes the Chamfer Distance between them and considers the prediction as true-positive only if the distance is less than a specified threshold, which is set to [0.2,0.5,1.0]m in our experiment. Note the overall AP metric is obtained by averaging across three thresholds.

Citation

@misc{qiao2023endtoend,
  title={End-to-End Vectorized HD-map Construction with Piecewise Bezier Curve},
  author={Limeng Qiao et al. (2023)},
  year={2023},
  note={arXiv:2306.09700}
}

• arXiv: 2306.09700