Monodepth Eval

monodepth-eval

Deconstructing Self-Supervised Monocular Reconstruction: The Design Decisions that Matter — Spencer et al. (2022) (arXiv:2208.01489, 2022)

What this evaluates

Evaluates the accuracy and generalization of self-supervised monocular depth estimation models across image-based, pointcloud-based, and edge-based metrics on automotive and diverse natural scenes.

Datasets

• Kitti Eigen (KE split) — total ?; splits: test (-1)
• Kitti Eigen-Benchmark (KEB split) — total ?; splits: test (-1)
• SYNS-Patches — total 1175; splits: test (1175)

Metrics

• AbsRel (primary) — range: other
  • Mean absolute relative error: (1/N) * sum(|pred - gt| / gt). Lower is better.
• δ < 1.25^1 (primary) — range: percent
  • Percentage of pixels where max(pred/gt, gt/pred) < 1.25. Higher is better.
• F-Score (pointcloud) (primary) — range: percent
  • 3D pointcloud-based F-Score computed from reconstructed depth maps. Higher is better.
• Chamfer — range: other
  • Chamfer distance between predicted and ground truth pointclouds. Lower is better.
• MAE — range: other
  • Mean absolute error in meters. Lower is better.

Input / output format

Input: Monocular video frames or stereo image pairs.

Output: Predicted depth map per input image.

Scoring recipe

1. Align predictions to ground truth: use per-image median scaling for monocular methods, fixed scaling factor for stereo methods.
2. Cap depth predictions at 100 meters.
3. Compute metrics (AbsRel, δ<1.25^1, F-Score, Chamfer, etc.) on aligned predictions vs. ground truth.
4. Average metrics over the dataset.

Common pitfalls

• Using outdated ground truth (Eigen & Fergus 2015) instead of corrected depth maps (Uhrig et al. 2018) for KEB.
• Applying border cropping or stereo-blending post-processing, which the authors explicitly omit.
• Using a 50m or 80m depth cap instead of the specified 100m cap.
• Applying fixed scaling to monocular predictions instead of per-image median scaling.

Evidence (verbatim from paper)

We cap the maximum depth to 100 meters (compared to the common $50\mathrm{m}$ (Garg et al., 2016) or $80\mathrm{m}$ (Zhou et al., 2017)) and omit border cropping (Garg et al., 2016) & stereo-blending post-processing (Godard et al., 2017). Again, we show the rank ordering based on image-based (AbsRel), pointcloud-based (F-Score) and edge-based (F-Score) metrics. Monocular-supervised (M) methods instead apply per-image median scaling to align the prediction and ground-truth.

Citation

@misc{spencer2022deconstructing,
  title={Deconstructing Self-Supervised Monocular Reconstruction: The Design Decisions that Matter},
  author={Spencer et al. (2022)},
  year={2022},
  note={arXiv:2208.01489}
}

• arXiv: 2208.01489