qhjqhj00 optical-flow-kitti-sintel-eval
Optical Flow Requires Multiple Strategies (but only one network) — Schuster et al. (2016) (arXiv:1611.05607, 2016)
What this evaluates
This evaluation protocol measures the accuracy of predicted optical flow fields against ground truth displacement vectors across varying motion magnitudes and occlusion conditions. It probes a model's ability to handle both small, fine-grained movements and large, robust displacements using standard video sequence benchmarks.
Datasets
- KITTI2012 — total ?; splits: train (-1), test (-1)
- KITTI2015 — total ?; splits: train (-1), test (-1)
- MPI-Sintel — total ?; splits: test (-1)
Metrics
Out-Noc(primary) — range: percent- Percentage of pixels with Euclidean error greater than 3 pixels, calculated only over non-occluded pixels.
Fl-all— range: percent- Percentage of outlier pixels (Euclidean error greater than 3 pixels) across all image regions.
EPE(primary) — range: pixels- End-point error averaged over all pixels (or restricted to specific displacement ranges like s0-10 or s40+).
Fl— range: percent- Percentage of pixels with an error value larger than 120.
Input / output format
Input: Paired images (or image patches) representing consecutive frames in a video sequence.
Output: Dense optical flow field (2D displacement vectors for each pixel or patch).
Scoring recipe
def compute_metrics(pred_flow, gt_flow, mask=None):
err = np.sqrt(np.sum((pred_flow - gt_flow)**2, axis=-1))
outlier_kitti = (err > 3.0).astype(float)
if mask is not None: outlier_kitti = outlier_kitti[mask]
out_noc = np.mean(outlier_kitti) * 100
fl_sintel = np.mean((err > 120).astype(float)) * 100
epe = np.mean(err)
return {'Out-Noc': out_noc, 'Fl': fl_sintel, 'EPE': epe}
Common pitfalls
- KITTI's Out-Noc metric explicitly excludes occluded pixels; reporting overall error without masking will overstate performance.
- MPI-Sintel's Fl metric uses a much higher outlier threshold (120 pixels) than KITTI (3 pixels), requiring separate thresholding logic.
- EPE is often reported globally, but the protocol emphasizes analyzing it within specific displacement ranges (e.g., s0-10, s40+) to evaluate multi-strategy behavior.
Evidence (verbatim from paper)
Out-Noc is the percentage of pixels with euclidean error $>$ 3 pixels out of the non-occluded pixels. Fl-all is the percentage of outliers (pixels with euclidean error $>$ 3 pixels). The EPE (end-point-error) is averaged over all the pixels and the two right columns contain only the EPE of pixels within the displacement range mentioned in the title. The Fl column presents an evaluation of the the outlier percentage, which, although not provided by this benchmark, was calculated from the error figures presented for each scene that have higher pixel values for larger errors. Fl is the percentage of pixels with a value larger than 120.
Citation
@misc{schuster2016optical,
title={Optical Flow Requires Multiple Strategies (but only one network)},
author={Schuster et al. (2016)},
year={2016},
note={arXiv:1611.05607}
}
- arXiv: 1611.05607