Voldor Eval

voldor-eval

VOLDOR: Visual Odometry from Log-logistic Dense Optical flow Residuals — Min et al. (2021) (arXiv:2104.06789, 2021)

What this evaluates

Evaluates monocular visual odometry accuracy and depth estimation quality on urban/highway driving sequences and indoor environments. Probes robustness to non-Gaussian optical flow noise and scale ambiguity without relying on hand-crafted features or loop closure.

Datasets

• KITTI odometry benchmark — total ?; splits: train (-1), test (-1)
• KITTI stereo benchmark — total ?; splits: train (-1), test (-1)
• TUM RGB-D dataset — total ?; splits: test (-1)

Metrics

• Trans. error (%) (primary) — range: percent
  • Percentage of total translation distance traveled incorrectly, averaged over sub-sequences of 100–800 m length.
• Rot. error (deg/m) (primary) — range: other
  • Total rotation angle in degrees divided by total translation distance in meters, averaged over sub-sequences.
• EPE (px) — range: other
  • Mean End-Point Error between predicted and ground-truth disparity/depth in pixels.
• bg-outlier (%) — range: percent
  • Percentage of background pixels where disparity error exceeds 3 px or 5%.
• Translation RMSE (m) — range: other
  • Root mean square error of the translation trajectory after rigid alignment to ground truth over 6-frame segments.

Input / output format

Input: Sequential RGB image pairs and externally estimated dense optical flow fields (e.g., from PWC-Net).

Output: 6-DoF camera pose transformations, per-pixel depth maps, and pixel-wise rigidness/confidence probabilities.

Scoring recipe

def evaluate_kitti(pred_poses, gt_poses):
    trans_err, rot_err = 0.0, 0.0
    for seq in sequences:
        for subseq in sliding_window(seq, 100, 100):
            trans_err += trajectory_error(pred_poses[subseq], gt_poses[subseq])
            rot_err += rotation_error(pred_poses[subseq], gt_poses[subseq])
    return trans_err / N, rot_err / N

def evaluate_tum(pred_poses, gt_poses):
    rmse_vals = []
    for seg in sliding_window(dataset, 6):
        aligned_gt = align_to_gt(gt_poses[seg])
        rmse_vals.append(translation_rmse(pred_poses[seg], aligned_gt))
    return mean(rmse_vals)

Common pitfalls

• Scale ambiguity: Monocular output requires explicit ground-plane height estimation or known camera height; unaligned scales invalidate error metrics.
• Rigidness threshold sensitivity: Depth map density and accuracy trade off sharply based on the W^j threshold, affecting EPE and outlier rates.
• Optical flow dependency: Performance is tightly coupled to the external flow estimator (e.g., PWC-Net); results degrade with noisier flow inputs.

Evidence (verbatim from paper)

The translation and rotation errors are averaged over all sub-sequences of length from 100 meters to 800 meters with 100 meter steps. ... In all instances, we rigidly align trajectories to groundtruth for segments with 6 frames and estimate mean translation RMSE of all segments.

Citation

@misc{min2021voldor,
  title={VOLDOR: Visual Odometry from Log-logistic Dense Optical flow Residuals},
  author={Min et al. (2021)},
  year={2021},
  note={arXiv:2104.06789}
}

• arXiv: 2104.06789