Kitti Depth Eval

kitti-depth-eval

SUB-Depth: Self-distillation and Uncertainty Boosting Self-supervised Monocular Depth Estimation — Zhou et al. (2021) (arXiv:2111.09692, 2021)

What this evaluates

Evaluates self-supervised monocular depth estimation models on outdoor driving scenes, measuring both the geometric accuracy of predicted depth maps and the reliability of associated uncertainty estimates.

Datasets

• KITTI — total ?; splits: train (39810), val (4424)

Metrics

• Abs Rel (primary) — range: [0, 1]
  • Mean absolute relative error: (1/N) Σ |d_pred - d_gt| / d_gt. Lower is better.
• Sq Rel — range: [0, 1]
  • Mean squared relative error: (1/N) Σ (d_pred - d_gt)² / d_gt. Lower is better.
• RMSE — range: [0, inf)
  • Root mean squared error: sqrt((1/N) Σ (d_pred - d_gt)²). Lower is better.
• RMSE log — range: [0, inf)
  • Root mean squared error of log-depth: sqrt((1/N) Σ (log d_pred - log d_gt)²). Lower is better.
• δ1 — range: [0, 1]
  • Accuracy threshold: (1/N) Σ I(max(d_pred/d_gt, d_gt/d_pred) < 1.25). Higher is better.
• δ2 — range: [0, 1]
  • Accuracy threshold: (1/N) Σ I(max(d_pred/d_gt, d_gt/d_pred) < 1.25²). Higher is better.
• δ3 — range: [0, 1]
  • Accuracy threshold: (1/N) Σ I(max(d_pred/d_gt, d_gt/d_pred) < 1.25³). Higher is better.
• AUSE — range: [0, 1]
  • Area Under the Sparsification Error curve. Measures how quickly uncertainty estimates correlate with actual error. Lower is better.
• AURG — range: [0, 1]
  • Area Under the Random Gain curve. Measures uncertainty calibration quality. Higher is better.

Input / output format

Input: RGB image frames resized to 640×192. Monocular frame triplets with a universal camera intrinsic matrix.

Output: Predicted dense depth map and per-pixel depth uncertainty map.

Scoring recipe

def compute_depth_metrics(pred, gt):
    valid = gt > 0
    p, g = pred[valid], gt[valid]
    abs_rel = np.mean(np.abs(p - g) / g)
    sq_rel = np.mean(((p - g) ** 2) / g)
    rmse = np.sqrt(np.mean((p - g) ** 2))
    rmse_log = np.sqrt(np.mean((np.log(p) - np.log(g)) ** 2))
    thresh = np.maximum((p / g), (g / p))
    d1 = np.mean(thresh < 1.25)
    d2 = np.mean(thresh < 1.25 ** 2)
    d3 = np.mean(thresh < 1.25 ** 3)
    return abs_rel, sq_rel, rmse, rmse_log, d1, d2, d3

Common pitfalls

• Confusing the Eigen split with other KITTI splits (e.g., Geiger) changes train/val/test sizes.
• AUSE and AURG have opposite optimization directions: lower AUSE is better, higher AURG is better.
• Photometric loss weighting is often confused with uncertainty weighting in ablation studies; the paper explicitly compares 1:1 fixed weighting vs. learned uncertainty weighting.

Evidence (verbatim from paper)

Depth metrics described by Eigen [9] are the most common used metrics for evaluating depth estimation accuracy. They include four error metrics: the Absolute Relative Error (Abs Rel), Squared Relative Error (Sq Rel), Root Mean Squared Error (RMSE), and the log of RMSE; accuracy metric: δ1, δ2, δ3.

Citation

@misc{zhou2021subdepth,
  title={SUB-Depth: Self-distillation and Uncertainty Boosting Self-supervised Monocular Depth Estimation},
  author={Zhou et al. (2021)},
  year={2021},
  note={arXiv:2111.09692}
}

• arXiv: 2111.09692