Blinkflow Eval

blinkflow-eval

BlinkFlow: A Dataset to Push the Limits of Event-based Optical Flow Estimation — Li et al. (2023) (arXiv:2303.07716, 2023)

What this evaluates

Evaluates the accuracy and robustness of event-based optical flow estimation models on complex scenes with dynamic objects, occlusions, and high-frequency motion. It measures how well models generalize to unseen scenarios and handle fine-grained flow details compared to traditional rigid/static scene benchmarks.

Datasets

• BlinkFlow — total 33000; splits: train (33000), test (-1)
• DSEC — total ?; splits: train (-1), test (-1)
• MVSEC — total ?; splits: train (-1), test (-1)

Metrics

• AEE (primary) — range: pixels
  • Average Euclidean distance between predicted and ground truth optical flow vectors per pixel.
• Outlier — range: percent
  • Percentage of pixels where endpoint error exceeds both 3 pixels and 5% of the ground truth flow magnitude.
• AE — range: degrees
  • Angle in degrees between normalized predicted and normalized ground truth flow vectors.
• N-PE — range: percent
  • Percentage of pixels with flow error magnitude exceeding N pixels (computed for N=1, 2, 3).

Input / output format

Input: Event stream data (and optionally corresponding frames) for consecutive time steps.

Output: 2D optical flow field (u, v vectors) for each pixel in the frame.

Scoring recipe

def compute_metrics(pred_flow, gt_flow):
    ee = np.sqrt((pred_flow[:,:,0] - gt_flow[:,:,0])**2 + (pred_flow[:,:,1] - gt_flow[:,:,1])**2)
    aee = np.mean(ee)
    mag_gt = np.sqrt(gt_flow[:,:,0]**2 + gt_flow[:,:,1]**2)
    outlier = np.mean((ee > 3) & (ee > 0.05 * mag_gt)) * 100
    norm_pred = pred_flow / (np.linalg.norm(pred_flow, axis=2, keepdims=True) + 1e-6)
    norm_gt = gt_flow / (np.linalg.norm(gt_flow, axis=2, keepdims=True) + 1e-6)
    ae = np.degrees(np.arccos(np.clip(np.sum(norm_pred * norm_gt, axis=2), -1.0, 1.0)))
    n_pe = {f"{n}-PE": np.mean(ee > n) * 100 for n in [1, 2, 3]}
    return {"AEE": aee, "Outlier": outlier, "AE": ae, **n_pe}

Common pitfalls

• DSEC and MVSEC ground truth is computed via post-processing (motion field multiplied by time interval dt), which introduces bias and inaccuracies compared to BlinkFlow's direct renderer ground truth.
• Models easily overfit to DSEC/MVSEC training data (outdoor driving scenes), leading to severe performance degradation on indoor or complex dynamic scenes.
• The Outlier metric uses a dual threshold (>3 pixels AND >5% of magnitude), not just a fixed pixel threshold.

Evidence (verbatim from paper)

We report the optical flow accuracy in terms of the Average Endpoint Error (AEE), outlier, Angular Error (AE) and N-pixel erros (N-PE). AEE measures the Euclidean distance between the predicted flow and the ground truth. Outlier is the percentage of points with endpoint error greater than 3 pixels and 5% of the magnitude. AE measures the angle between the normalized optical flow predictions and normalized ground truth. N-PE indicates the percentage of flow errors higher than N pixels in magnitude and we compute 3-PE, 2-PE, 1-PE for flow outliers analysis from coarse to fine.

Citation

@misc{li2023blinkflow,
  title={BlinkFlow: A Dataset to Push the Limits of Event-based Optical Flow Estimation},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2303.07716}
}

• arXiv: 2303.07716