qhjqhj00 kitti-fc-eval
Benchmarking the Robustness of Optical Flow Estimation to Corruptions — Yi et al. (2024) (arXiv:2411.14865, 2024)
What this evaluates
Evaluates the robustness of optical flow estimation models when subjected to various digital, illumination, weather, noise, and blur corruptions. It measures both absolute performance degradation and relative robustness compared to clean data across in-domain and out-of-domain training settings.
Datasets
- KITTI-FC — total ?; splits: train (-1), test (-1); repo https://github.com/ZhonghuaYi/optical_flow_robustness_benchmark
Metrics
EPE(primary) — range: pixels- End-Point Error: the mean Euclidean distance between predicted and ground-truth flow vectors across all valid pixels.
CRE— range: pixels- Corruption Robustness Error: quantifies absolute robustness by measuring the increase in EPE under corruption compared to clean data.
CREr— range: ratio or percent- Relative CRE: quantifies relative robustness drop, typically calculated as the ratio or percentage change in EPE degradation between evaluation settings (e.g., OOD vs ID).
Input / output format
Input: Paired image frames (or single frame with applied corruption) for optical flow estimation.
Output: 2D optical flow field (horizontal and vertical displacement vectors per pixel).
Scoring recipe
def compute_epe(pred_flow, gt_flow):
return np.mean(np.sqrt(np.sum((pred_flow - gt_flow)**2, axis=-1)))
def compute_cre(epe_clean, epe_corrupted):
return epe_corrupted - epe_clean
def compute_crer(epe_clean_id, epe_corrupted_id, epe_clean_ood, epe_corrupted_ood):
deg_id = compute_cre(epe_clean_id, epe_corrupted_id)
deg_ood = compute_cre(epe_clean_ood, epe_corrupted_ood)
return (deg_id - deg_ood) / deg_ood if deg_ood > 0 else 0.0
Common pitfalls
- Confusing Out-Of-Domain (OOD) evaluation (trained on FlyingThings/Sintel/HD1K) with In-Domain (ID) evaluation (fine-tuned on KITTI-FC train split).
- Overlooking that unsupervised methods are only evaluated in the ID setting, while supervised models are tested in both.
- Failing to apply the exact 24 corruptions (7 temporal, 17 classical) including the upgraded PSF blur simulation method.
Evidence (verbatim from paper)
We first report the EPE of all 29 model variants to give the optical flow estimation performance in Tab. 2, then discuss the absolute robustness CRE and relative robustness CREr in Fig. 4 for deep research on optical flow robustness.
Citation
@misc{yi2024benchmarking,
title={Benchmarking the Robustness of Optical Flow Estimation to Corruptions},
author={Yi et al. (2024)},
year={2024},
note={arXiv:2411.14865}
}
- arXiv: 2411.14865