Kitti Eigen Depth Eval

kitti-eigen-depth-eval

Digging Into Self-Supervised Monocular Depth Estimation — Godard et al. (2018) (arXiv:1806.01260, 2018)

What this evaluates

Evaluates the accuracy of self-supervised monocular depth estimation models on urban driving scenes. It probes the model's ability to predict per-pixel depth from a single image or video sequence, handling occlusions, moving objects, and scale ambiguity.

Datasets

• KITTI 2015 (Eigen split) — total ?; splits: train (39810), val (4424), test (-1)
• Make3D — total ?; splits: test (-1)

Metrics

• Abs Rel (primary) — range: other
  • Mean absolute relative error: average of |d_pred - d_gt| / d_gt across valid pixels.
• δ < 1.25 (primary) — range: [0, 1]
  • Fraction of pixels where max(d_pred/d_gt, d_gt/d_pred) < 1.25.
• RMSE — range: other
  • Root mean squared error: sqrt(mean((d_pred - d_gt)^2)) across valid pixels.

Input / output format

Input: Single RGB image (or sequence of frames for video models) typically at 512×160 or 1024×320 resolution.

Output: Per-pixel depth map (H×W matrix of float values representing distance in meters).

Scoring recipe

def compute_metrics(pred_depth, gt_depth):
    # Apply per-image median scaling for monocular models
    scale = np.median(gt_depth) / np.median(pred_depth)
    pred_scaled = pred_depth * scale
    # Filter invalid pixels (depth > 80m or <= 0)
    valid = (gt_depth > 0) & (gt_depth <= 80) & (pred_scaled > 0)
    d_pred = pred_scaled[valid]
    d_gt = gt_depth[valid]
    abs_rel = np.mean(np.abs(d_pred - d_gt) / d_gt)
    rmse = np.sqrt(np.mean((d_pred - d_gt)**2))
    delta = np.mean(np.maximum(d_pred/d_gt, d_gt/d_pred) < 1.25)
    return abs_rel, rmse, delta

Common pitfalls

• Per-image median scaling must be applied for monocular models but NOT for stereo-supervised models, as scale is recovered from the known baseline during training.
• Results are reported without post-processing; applying standard post-processing significantly improves scores but is not the default reported protocol.
• The model assumes Lambertian surfaces and fails on moving objects or non-static scenes, which requires auto-masking during training but isn't handled during evaluation.

Evidence (verbatim from paper)

We evaluate our models, named Monodepth2, on the KITTI 2015 stereo dataset [13], to allow comparison with previously published monocular methods. For our monocular models, we report results using the per-image median ground truth scaling introduced by [76]. See also supplementary material Section D.2 for results where we apply a single median scaling to the whole test set, instead of scaling each image independently. Table 1. Quantitative results. Comparison of our method to existing methods on KITTI 2015 [13] using the Eigen split. ... Abs Rel ... Sq Rel ... RMSE ... RMSE log ... δ < 1.25 ...

Citation

@misc{godard2018digging,
  title={Digging Into Self-Supervised Monocular Depth Estimation},
  author={Godard et al. (2018)},
  year={2018},
  note={arXiv:1806.01260}
}

• arXiv: 1806.01260