Spring Benchmark Eval

spring-benchmark-eval

Spring: A High-Resolution High-Detail Dataset and Benchmark for Scene Flow, Optical Flow and Stereo — Mehl et al. (2023) (arXiv:2303.01943, 2023)

What this evaluates

This benchmark evaluates the ability of computer vision models to estimate dense scene flow, optical flow, and stereo disparity at ultra-high resolutions with fine structural details. It specifically probes how well methods handle high-frequency textures, non-rigid motion, unmatched regions, and sky areas where traditional benchmarks often lack detail.

Datasets

• Spring — total ?; splits: train (5000), test (1000)

Metrics

• 1px outlier rate (primary) — range: percent
  • Percentage of pixels where the estimated value (disparity or flow) deviates by more than 1 pixel from the ground truth. For scene flow, a union variant is used, marking a pixel as an outlier if any of the reference disparity, target disparity, or optical flow estimates deviate by >1px.
• EPE — range: pixels
  • Average Endpoint Error: the mean Euclidean distance between predicted and ground truth optical flow vectors across all pixels.
• Fl error — range: percent
  • KITTI Fl outlier rate: percentage of pixels where the flow error exceeds 3 pixels or 5% of the ground truth magnitude.
• WAUC — range: percent
  • Weighted Area Under Curve metric for optical flow evaluation, weighting errors by displacement magnitude.
• Abs error — range: pixels
  • Absolute disparity error: mean absolute difference between predicted and ground truth disparity values.
• D1 error — range: percent
  • KITTI D1 outlier rate for stereo: percentage of pixels in non-occluded regions where disparity error exceeds 3 pixels or 5%.
• SF error — range: percent
  • KITTI SF outlier rate for scene flow: percentage of pixels where the 3D flow error exceeds 0.5 pixels or 5%.

Input / output format

Input: High-resolution image pairs (for stereo) or consecutive frames (for optical/scene flow) from 47 synthetic sequences.

Output: Dense prediction maps (disparity/flow fields) submitted as subsampled files via a public benchmark website.

Scoring recipe

def compute_1px_outlier(pred, gt):
    error = np.abs(pred - gt)
    outlier_mask = error > 1.0
    return 100.0 * np.mean(outlier_mask)

def compute_union_1px_outlier(pred_d1, pred_d2, pred_flow, gt_d1, gt_d2, gt_flow):
    err_d1 = np.abs(pred_d1 - gt_d1) > 1.0
    err_d2 = np.abs(pred_d2 - gt_d2) > 1.0
    err_flow = np.abs(pred_flow - gt_flow) > 1.0
    return 100.0 * np.mean(err_d1 | err_d2 | err_flow)

def compute_epe(pred_flow, gt_flow):
    return np.mean(np.sqrt(np.sum((pred_flow - gt_flow)**2, axis=-1)))

Common pitfalls

• The benchmark uses a subsampling strategy for test submissions to reduce file size; while rankings remain stable, absolute metric values differ from full-evaluation results.
• Stereo methods often assume strictly positive disparity, causing poor performance on sky regions (zero disparity/infinite depth) which are explicitly included in the test set.
• Ground truth is generated at double resolution (4 values per pixel); evaluation selects the closest GT value, making the metric more permissive for thin structures like hair.

Evidence (verbatim from paper)

Considering our high-accuracy data, we adapt the evaluation to the 1px outlier rate. For reference disparity, target disparity and optical flow, the 1px outlier rate defines the percentage of pixels that deviate more than 1px from the ground truth. Following [30], we also employ a union 1px error as the main scene flow measure that defines the percentage of pixels where any of the estimated reference disparity, target disparity and optical flow values deviates more than 1px from the ground truth.

Citation

@misc{mehl2023spring,
  title={Spring: A High-Resolution High-Detail Dataset and Benchmark for Scene Flow, Optical Flow and Stereo},
  author={Mehl et al. (2023)},
  year={2023},
  note={arXiv:2303.01943}
}

• arXiv: 2303.01943