Waymo Open Dataset Eval

waymo-open-dataset-eval

Scalability in Perception for Autonomous Driving: Waymo Open Dataset — Pei Sun et al. (2019) (arXiv:1912.04838, 2019)

What this evaluates

This benchmark evaluates 3D and 2D object detection, as well as multi-object tracking, for autonomous driving perception. It probes a model's ability to accurately localize vehicles and pedestrians using synchronized LiDAR and camera data, while measuring robustness to geographic domain shifts and varying training data scales.

Datasets

• Waymo Open Dataset — total 1150; splits: train (-1), val (-1), test (150)

Metrics

• APH (primary) — range: percent
  • Average Precision with Heading. Computes AP across confidence thresholds for 3D bounding boxes, requiring both IoU overlap and heading angle alignment. IoU thresholds are 0.7 for vehicles and 0.5 for pedestrians. Difficulty is split into LEVEL_1 (≥5 LiDAR points, not labeled hard) and LEVEL_2 (≤5 points or labeled hard).
• MOTA — range: percent
  • Multi-Object Tracking Accuracy. Measures tracking performance by penalizing false positives, misses, and identity switches relative to the total number of ground truth frames across the sequence.

Input / output format

Input: Single-frame LiDAR point clouds (for 3D detection) or synchronized camera images (for 2D detection). For tracking, sequential frames with prior detection outputs.

Output: 3D bounding boxes (cx, cy, cz, w, l, h, heading) with class and confidence score; or 2D bounding boxes with class and confidence; or track IDs with state (Live/Pending/Dead) and motion parameters.

Scoring recipe

def compute_3d_detection_ap(pred_boxes, gt_boxes, iou_thresh):
    matches = []
    for pred in sorted(pred_boxes, key=lambda x: x.confidence, reverse=True):
        best_gt = max(gt_boxes, key=lambda g: iou(pred.box, g.box), default=None)
        if best_gt and iou(pred.box, best_gt.box) >= iou_thresh:
            matches.append(best_gt)
            gt_boxes.remove(best_gt)
    # AP is computed by sorting matches by confidence and calculating precision-recall curve
    return average_precision(matches)

Common pitfalls

• IoU thresholds differ by object class (0.7 for vehicles, 0.5 for pedestrians) and must be applied separately during evaluation.
• Difficulty levels (LEVEL_1 vs LEVEL_2) are determined by LiDAR point density and labeler annotations, not just distance or occlusion.
• The test set is hidden and not publicly available for direct submission; evaluation is typically done via a centralized server or by using the validation set for domain gap studies.

Evidence (verbatim from paper)

For the 3D LiDAR-based vehicle object detector, we observed an APH reduction of 8.0 when training on SF and evaluating on SUB compared with training on SUB and evaluating on SUB, and an APH reduction of 7.6 when training on SUB and evaluating on SF compared with training on SF and evaluating on SF.

Citation

@misc{sun2019waymo,
  title={Scalability in Perception for Autonomous Driving: Waymo Open Dataset},
  author={Pei Sun et al. (2019)},
  year={2019},
  note={arXiv:1912.04838}
}

• arXiv: 1912.04838