Far3det Eval

far3det-eval

Far3Det: Towards Far-Field 3D Detection — Gupta et al. (2022) (arXiv:2211.13858, 2022)

What this evaluates

Evaluates 3D object detection performance specifically in the far-field range (50-80m), highlighting the limitations of fixed-threshold metrics and sparse lidar data. It probes how well models detect distant objects using lidar, RGB, or fused modalities under adaptive distance-aware tolerance thresholds.

Datasets

• nuScenes — total ?; splits: val (-1)
• Far nuScenes — total ?; splits: val (-1)

Metrics

• 3D mAP (primary) — range: percent
  • Mean Average Precision computed per class (Car, Truck, Pedestrian) over two distance ranges (0-50m and 50-80m). Uses adaptive distance-aware IoU thresholds (linear or quadratic scaling with distance) instead of fixed thresholds to match predictions to ground truth.
• 3D mAP (Elliptical) — range: percent
  • Same as 3D mAP but uses an elliptical thresholding scheme that excludes close-by regions and scales tolerance based on object orientation and distance.

Input / output format

Input: Lidar point clouds (aggregated sweeps) and/or RGB images. Models output 3D bounding box coordinates and class labels.

Output: 3D cuboid coordinates and class labels per detected object.

Scoring recipe

def compute_far3det_mAP(preds, gts, dist_range, scheme='linear'):
    ap_scores = []
    for cls in ['car', 'truck', 'pedestrian']:
        cls_preds = [p for p in preds if p.cls == cls and dist_range[0] <= p.dist <= dist_range[1]]
        cls_gts = [g for g in gts if g.cls == cls and dist_range[0] <= g.dist <= dist_range[1]]
        # Compute AP using adaptive IoU threshold based on scheme
        ap = compute_AP_with_adaptive_iou(cls_preds, cls_gts, scheme)
        ap_scores.append(ap)
    return sum(ap_scores) / len(ap_scores) * 100

Common pitfalls

• Using fixed IoU/distance thresholds (e.g., 0.5m) for far-field evaluation unfairly penalizes image-based methods due to lidar sparsity.
• Ignoring zero-lidar-point objects in the ground truth inflates lidar-only mAP and hides far-field detection failures.
• Reporting a single aggregate mAP without splitting by distance range (0-50m vs 50-80m) masks the severe performance drop-off in the far field.

Evidence (verbatim from paper)

We evaluate the car, truck, and pedestrian mAP of lidar- (CenterPoint [39]) and image-based (FCOS3D [31]) models on 0-50m and 50-80m distance range. We use CP as an abbreviation for CenterPoint. Table 3 shows the mAP values for the 0-50m and 50-80m range on nuScenes validation set (other classes in appendix). We observe that the Far3Det mAP is lower for image-based method (column d) compared to lidar-based method when we use the default nuScenes thresholding scheme, however when we use our proposed linear and quadratic thresholding schemes (d & e), we observe that image-based method outperforms lidar-based method for cars and trucks.

Citation

@misc{gupta2022far3det,
  title={Far3Det: Towards Far-Field 3D Detection},
  author={Gupta et al. (2022)},
  year={2022},
  note={arXiv:2211.13858}
}

• arXiv: 2211.13858