Per Object Depth Estimation Eval

per-object-depth-estimation-eval

Depth Estimation Matters Most: Improving Per-Object Depth Estimation for Monocular 3D Detection and Tracking — Longlong Jing et al. (arXiv:2206.03666, 2022)

What this evaluates

Evaluates the accuracy of per-object depth estimation for vehicles in autonomous driving scenarios. It measures how well a model predicts the depth of individual objects given monocular RGB images and 2D bounding boxes.

Datasets

• Waymo Open Dataset — total 1000; splits: train (798), val (202)
• KITTI Detection Dataset — total 7480; splits: train (3712), test (3768)
• KITTI MOT Dataset — total 19103; splits: train (8008), test (11095)

Metrics

• delta<1.25 (primary) — range: percent
  • Percentage of object pixels where max(d_pred/d_gt, d_gt/d_pred) < 1.25. Higher is better.
• Abs Rel — range: other
  • Mean of |d_pred - d_gt| / d_gt across all object pixels. Lower is better.
• Sq Rel — range: other
  • Mean of (d_pred - d_gt)^2 / d_gt across all object pixels. Lower is better.
• RMSE — range: other
  • Root-mean-square error of depth predictions: sqrt(mean((d_pred - d_gt)^2)). Lower is better.
• RMSE_log — range: other
  • Root-mean-square error of log10 depth predictions: sqrt(mean((log10(d_pred) - log10(d_gt))^2)). Lower is better.

Input / output format

Input: Monocular RGB image crop of the object, 2D bounding box coordinates, and optionally pseudo-LiDAR features or temporal tracklet frames.

Output: Per-object depth map or depth values for the object pixels.

Scoring recipe

def compute_depth_metrics(pred, gt):
    abs_rel = np.mean(np.abs(pred - gt) / gt)
    sq_rel = np.mean(((pred - gt) ** 2) / gt)
    rmse = np.sqrt(np.mean((pred - gt) ** 2))
    rmse_log = np.sqrt(np.mean((np.log10(pred) - np.log10(gt)) ** 2))
    delta = np.mean(np.maximum(pred / gt, gt / pred) < 1.25) * 100
    return {'Abs Rel': abs_rel, 'Sq Rel': sq_rel, 'RMSE': rmse, 'RMSE_log': rmse_log, 'delta<1.25': delta}

Common pitfalls

• Evaluating with predicted 2D boxes instead of ground truth boxes, which conflates detection error with depth estimation error.
• Fusing temporal features without compensating for camera ego-motion, leading to misaligned depth predictions.
• Ignoring the vehicle-only focus, as the benchmark is specifically tuned for the vehicle class in autonomous driving.

Evidence (verbatim from paper)

Following the existing state-of-the-art per-object depth estimation benchmark proposed by [[27]], five standard metrics including average relative error (Abs Rel), squared relative error (Sq Rel), root-mean-square error (RMSE), average (log10) error (RMSE_log), and threshold accuracy (delta_i) are used for evaluation.

Citation

@misc{jing2022depthestimation,
  title={Depth Estimation Matters Most: Improving Per-Object Depth Estimation for Monocular 3D Detection and Tracking},
  author={Longlong Jing et al.},
  year={2022},
  note={arXiv:2206.03666}
}

• arXiv: 2206.03666