Rsrd Dense Eval

rsrd-dense-eval

RSRD: A Road Surface Reconstruction Dataset and Benchmark for Safe and Comfortable Autonomous Driving — Zhao et al. (2023) (arXiv:2310.02262, 2023)

What this evaluates

Evaluates the ability of monocular depth estimation and stereo matching models to reconstruct fine-grained road surface profiles and disparities from high-resolution images. It probes the models' accuracy in capturing micro-level road textures and handling near-to-far distance variations under diverse dynamic conditions.

Datasets

• RSRD-dense — total 5500; splits: train (2500), test (300)
• RSRD-sparse — total ?; splits: (unstated)

Metrics

• Abs Rel (primary) — range: other
  • Mean absolute relative error: (1/N) * Σ |d_pred - d_gt| / d_gt. Lower is better.
• RMSE — range: other
  • Root mean square error: sqrt((1/N) * Σ (d_pred - d_gt)^2). Lower is better.
• δ < 1.25 — range: [0, 1]
  • Percentage of pixels where max(d_pred/d_gt, d_gt/d_pred) < 1.25. Higher is better.
• EPE — range: other
  • End Point Error: average absolute disparity error across all pixels. Lower is better.
• >1 px (%) — range: percent
  • Percentage of pixels with absolute disparity error > 1 pixel. Lower is better.
• >3 px (%) — range: percent
  • Percentage of pixels with absolute disparity error > 3 pixels. Lower is better.

Input / output format

Input: Left camera RGB images (full-resolution for monocular depth) or stereo image pairs (center-cropped for stereo matching).

Output: Predicted depth maps (meters) for monocular tasks, or predicted disparity maps (pixels) for stereo tasks.

Scoring recipe

def compute_depth_metrics(pred, gt):
    abs_rel = np.mean(np.abs(pred - gt) / gt)
    rmse = np.sqrt(np.mean((pred - gt) ** 2))
    delta = np.mean(np.maximum(pred / gt, gt / pred) < 1.25)
    return {'Abs Rel': abs_rel, 'RMSE': rmse, 'δ<1.25': delta}

def compute_stereo_metrics(pred_disp, gt_disp):
    err = np.abs(pred_disp - gt_disp)
    epe = np.mean(err)
    p1 = np.mean(err > 1) * 100
    p3 = np.mean(err > 3) * 100
    return {'EPE': epe, '>1 px (%)': p1, '>3 px (%)': p3}

Common pitfalls

• Do not use data augmentations (e.g., interpolation-based transforms) as they introduce noise in depth/disparity estimation.
• Evaluate near vs. far distances separately by splitting the image into upper and lower halves, as road texture coarseness significantly impacts far-distance accuracy.
• Respect the maximum depth (14m) and disparity (128px) constraints during inference to match the benchmark setup.

Evidence (verbatim from paper)

We also adopt the commonly utilized metrics in-depth estimation to evaluate the models, as shown in Table[II]... We evaluate the model performance with the end point error (EPE) calculated as the average absolute disparity error. The error ratios bigger than one and three pixels are also presented in Table [IV].

Citation

@misc{zhao2023rsrd,
  title={RSRD: A Road Surface Reconstruction Dataset and Benchmark for Safe and Comfortable Autonomous Driving},
  author={Zhao et al. (2023)},
  year={2023},
  note={arXiv:2310.02262}
}

• arXiv: 2310.02262