Robustspring Eval

robustspring-eval

RobustSpring: Benchmarking Robustness to Image Corruptions for Optical Flow, Scene Flow and Stereo — Schmalfuss et al. (2025) (arXiv:2505.09368, 2025)

What this evaluates

Evaluates the robustness of dense correspondence models (optical flow, scene flow, stereo) to 20 types of image corruptions by measuring the divergence between predictions on clean images and predictions on corrupted images.

Datasets

• Spring — total ?; splits: test (-1)

Metrics

• R^c_EPE (primary) — range: other
  • End-Point Error between the model's prediction on clean images and its prediction on corrupted images: $R^c_{\text{EPE}} = \frac{1}{N} \sum_{i} | \hat{y}{\text{clean}, i} - \hat{y}{\text{corrupted}, i} |2$. Lower values indicate higher robustness. Variants include $R^c{\text{1px}}$, $R^c_{\text{Abs}}$, $R^c_{\text{D1}}$, $R^c_{\text{D2}}$, and $R^c_{\text{Fl}}$ for different error thresholds.

Input / output format

Input: Corrupted image pairs (or video frames) with applied corruptions (noise, blur, weather, quality, color) and ground truth dense correspondences/disparities.

Output: Dense flow vectors, scene flow vectors, or disparity maps.

Scoring recipe

def compute_rc_epe(pred_clean, pred_corrupted):
    # pred_clean and pred_corrupted are dense flow/disparity maps
    error = np.linalg.norm(pred_clean - pred_corrupted, axis=-1)
    return np.mean(error)

Common pitfalls

• Ranking models using only the Average $R^c_{\text{EPE}}$ can misrepresent robustness compared to Median or Schulze rankings, as averages are highly sensitive to outlier corruptions like rain.
• Subsampling must exclude full-resolution 'Hero-frames'; otherwise, the 0.05% subsample will not align with full-dataset results.
• Weather and noise corruptions dominate the error scores, often masking performance differences on milder corruptions.

Evidence (verbatim from paper)

corruption robustness $R^{c}_{\text{EPE}}$ between clean and corrupted predictions. Low values indicate robust models

Citation

@misc{schmalfuss2025robustspring,
  title={RobustSpring: Benchmarking Robustness to Image Corruptions for Optical Flow, Scene Flow and Stereo},
  author={Schmalfuss et al. (2025)},
  year={2025},
  note={arXiv:2505.09368}
}

• arXiv: 2505.09368