Kitti Lidar Flow Eval

kitti-lidar-flow-eval

Hallucinating Dense Optical Flow from Sparse Lidar for Autonomous Vehicles — Victor Vaquero, Alberto Sanfeliu, Francesc Moreno-Noguer (2018) (arXiv:1808.10542, 2018)

What this evaluates

This evaluation probes a model's ability to estimate dense optical flow directly from sparse, noisy LiDAR range scans without using RGB images. It measures prediction accuracy against real-world ground truth flow maps and evaluates robustness to occlusions and foreground/background motion.

Datasets

• KITTI Tracking & Flow 2015 — total 19045; splits: train (17500), val (1455), test (90)

Metrics

• EPE (End-Point-Error) (primary) — range: pixels
  • Average Euclidean distance between predicted and ground-truth optical flow vectors across all pixels.
• Outlier Percentage — range: percent
  • Percentage of pixels where the EPE is less than 3 pixels or less than 5% of the ground-truth flow magnitude.

Input / output format

Input: Consecutive pairs of sparse LiDAR scans (Velodyne HDL-64) containing range and reflectivity values, formatted as 64x384 grids.

Output: Dense optical flow map with 2D displacement vectors per pixel, resolution 256x1224.

Scoring recipe

def compute_metrics(pred_flow, gt_flow):
    # pred_flow, gt_flow: (H, W, 2) arrays
    diff = pred_flow - gt_flow
    epe = np.mean(np.sqrt(np.sum(diff**2, axis=-1)))
    gt_mag = np.sqrt(np.sum(gt_flow**2, axis=-1))
    outlier = (np.sqrt(np.sum(diff**2, axis=-1)) < 3) | \
              (np.sqrt(np.sum(diff**2, axis=-1)) / (gt_mag + 1e-6) < 0.05)
    return epe, np.mean(outlier) * 100

Common pitfalls

• The test set is extremely small (90 pairs) because it requires matching RGB frames from KITTI Flow 2015 with LiDAR frames from the Tracking benchmark.
• Training uses pseudo-ground-truth flow generated by FlowNet2 on RGB images, but evaluation against real ground-truth is required for benchmark comparison.
• The outlier threshold uses an OR condition (<3px OR <5%), not an AND condition, which significantly changes the reported percentage.

Evidence (verbatim from paper)

A pixel is considered to be correctly estimated if the End-Point-Error (EPE) calculated as the averaged Euclidean distance between the prediction and the real ground-truth $G_{Test}$ is $<3$px or $<5$%. These measurements are averaged over background regions only, over foreground regions only, and over all ground truth pixels, which respectively are denoted in Table I as “Fl-BG”, “Fl-FG” and “All”.

Citation

@misc{vaquero2018hallucinating,
  title={Hallucinating Dense Optical Flow from Sparse Lidar for Autonomous Vehicles},
  author={Victor Vaquero, Alberto Sanfeliu, Francesc Moreno-Noguer (2018)},
  year={2018},
  note={arXiv:1808.10542}
}

• arXiv: 1808.10542