Optical Flow Estimation Eval

optical-flow-estimation-eval

LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation — Hui et al. (2018) (arXiv:1805.07036, 2018)

What this evaluates

Evaluates the accuracy of predicted optical flow fields against ground truth motion vectors between consecutive image frames. It probes a model's ability to estimate dense pixel-wise displacement in both synthetic cinematic scenes and real-world driving environments.

Datasets

• FlyingChairs — total ?; splits: test (-1)
• Sintel — total ?; splits: clean (-1), final (-1)
• KITTI12 — total ?; splits: test (-1)
• KITTI15 — total ?; splits: test (-1)
• Middlebury — total ?; splits: test (-1)

Metrics

• AEE (primary) — range: other
  • Average End-Point Error. Computed as the mean Euclidean distance between the predicted and ground truth flow vectors across all pixels: AEE = (1/N) * sum(||f_pred - f_gt||_2).
• Fl-all — range: percent
  • Percentage of outliers averaged over all pixels. A pixel is considered an inlier if its EPE is less than 3 pixels or less than 5% of the ground truth flow magnitude; Fl-all is the complement percentage.

Input / output format

Input: Pairs of consecutive frames (RGB images) from a video sequence.

Output: A dense optical flow field (H x W x 2 tensor) representing the horizontal and vertical displacement vectors for each pixel.

Scoring recipe

def compute_aee(pred_flow, gt_flow):
    epe = np.sqrt(np.sum((pred_flow - gt_flow)**2, axis=-1))
    return np.mean(epe)

def compute_fl_all(pred_flow, gt_flow):
    epe = np.sqrt(np.sum((pred_flow - gt_flow)**2, axis=-1))
    gt_mag = np.linalg.norm(gt_flow, axis=-1)
    inlier = (epe < 3.0) | (epe < 0.05 * gt_mag)
    return 100.0 * (1.0 - np.mean(inlier))

Common pitfalls

• The paper explicitly notes that results in parentheses are computed on training data and are 'not directly comparable to the others' (test set results).
• Runtime measurements vary significantly by framework (Torch vs Caffe); comparing speeds across models requires noting the implementation used.
• The outlier threshold for Fl-all uses a logical OR between 3px and 5% of flow magnitude; using only one threshold will yield different values.

Evidence (verbatim from paper)

Average end-point error (AEE) is reported. ... Fl-all: Percentage of outliers averaged over all pixels. Inliers are defined as EPE < 3 pixels or < 5%.

Citation

@misc{hui2018liteflownet,
  title={LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation},
  author={Hui et al. (2018)},
  year={2018},
  note={arXiv:1805.07036}
}

• arXiv: 1805.07036