Kitti Subt Vo Depth Eval

kitti-subt-vo-depth-eval

Unsupervised Deep Persistent Monocular Visual Odometry and Depth Estimation in Extreme Environments — Almalioglu et al. (2020) (arXiv:2011.00341, 2020)

What this evaluates

Evaluates unsupervised monocular visual odometry and depth estimation methods on challenging driving and subterranean environments. Probes the model's ability to predict consistent 6-DoF ego-motion and recover accurate depth maps without ground-truth supervision.

Datasets

• KITTI — total ?; splits: test (-1)
• DARPA Subterranean Challenge — total ?; splits: test (-1)

Metrics

• relative translation error ($t_{err}$) (primary) — range: percent
  • Relative translation error divided by distance travelled, averaged over trajectory segments of lengths {7, 14, 21, 28, 35} m. Predictions are aligned to ground truth via 7-DoF optimization minimizing ATE.
• relative rotation error ($r_{err}$) (primary) — range: percent
  • Relative rotation error divided by distance travelled, averaged over trajectory segments of lengths {7, 14, 21, 28, 35} m. Predictions are aligned to ground truth via 7-DoF optimization minimizing ATE.
• AbsRel — range: other
  • Mean absolute relative difference between predicted and ground truth depth: mean(|pred - gt| / gt).
• SqRel — range: other
  • Mean squared relative difference: mean((pred - gt)^2 / gt).
• RMS — range: other
  • Root mean square error between predicted and ground truth depth.
• RMSlog — range: other
  • Root mean square error between log-predicted and log-ground truth depth.
• δ<1.25 — range: percent
  • Percentage of pixels where max(pred/gt, gt/pred) < 1.25.
• δ<1.25² — range: percent
  • Percentage of pixels where max(pred/gt, gt/pred) < 1.25².
• δ<1.25³ — range: percent
  • Percentage of pixels where max(pred/gt, gt/pred) < 1.25³.

Input / output format

Input: Sequential RGB images resized to 416×256 pixels.

Output: Per-frame 6-DoF ego-motion (translation and rotation) and a monocular depth map.

Scoring recipe

# Pose evaluation
t_errs, r_errs = [], []
for seg_len in [7, 14, 21, 28, 35]:
    for seg in trajectory_segments(length=seg_len):
        aligned_pred = align_7dof(seg.predicted_poses, seg.gt_poses)
        rel_err = compute_relative_error(aligned_pred, seg.gt_poses)
        t_errs.append(rel_err.t_err)
        r_errs.append(rel_err.r_err)
t_err = mean(t_errs)
r_err = mean(r_errs)

# Depth evaluation (Eigen split)
absrel = mean(abs(pred_depth - gt_depth) / gt_depth)
sqrel = mean((pred_depth - gt_depth)**2 / gt_depth)
rms = sqrt(mean((pred_depth - gt_depth)**2))
rmslog = sqrt(mean((log(pred_depth) - log(gt_depth))**2))
acc1 = mean((pred_depth / gt_depth < 1.25) & (gt_depth / pred_depth < 1.25))
acc2 = mean((pred_depth / gt_depth < 1.25**2) & (gt_depth / pred_depth < 1.25**2))
acc3 = mean((pred_depth / gt_depth < 1.25**3) & (gt_depth / pred_depth < 1.25**3))

Common pitfalls

• Monocular pose predictions must be aligned to ground truth via 7-DoF optimization (minimizing ATE) before computing relative errors, as they lack real-world scale.
• Depth metrics are computed on the Eigen et al. crop/split of KITTI, not full-resolution images, which drastically changes absolute metric values.
• Relative pose errors are averaged across multiple segment lengths (7–35 m), not reported for a single fixed distance.

Evidence (verbatim from paper)

Thus, we show statistics for the relative translation and rotation error, divided by the distance travelled and averaged over the trajectory segments of lengths ${7,14,21,28,35}$ m over all sequences based on the shortest sequence. Here, the depth is evaluated on the Eigen et al. split of the raw KITTI dataset following the previous works.

Citation

@misc{almalioglu2020unsupervised,
  title={Unsupervised Deep Persistent Monocular Visual Odometry and Depth Estimation in Extreme Environments},
  author={Almalioglu et al. (2020)},
  year={2020},
  note={arXiv:2011.00341}
}

• arXiv: 2011.00341