Layeredflow Eval

layeredflow-eval

LayeredFlow: A Real-World Benchmark for Non-Lambertian Multi-Layer Optical Flow — Wen et al. (2024) (arXiv:2409.05688, 2024)

What this evaluates

Evaluates optical flow estimation on non-Lambertian surfaces (transparent, reflective, diffuse) and multi-layer scenes. It probes a model's ability to predict flow through transparent occluders and handle complex material properties without relying on test-time optimizations.

Datasets

• LayeredFlow — total 150000; splits: test (-1)

Metrics

• EPE (primary) — range: pixels
  • Average L2 distance between predicted and ground truth optical flow vectors across all evaluated pixels.
• bad-$\tau$ — range: percent
  • Percentage of pixels where the L2 error between predicted and ground truth flow exceeds a threshold $\tau$ (e.g., 1, 3, 5, or $\infty$ pixels).

Input / output format

Input: Pairs of consecutive frames from the LayeredFlow benchmark, downsampled to 540×960 resolution.

Output: Per-pixel 2D optical flow vectors (single vector for single-layer evaluation; multiple vectors per pixel for multi-layer evaluation).

Scoring recipe

def compute_epe(pred, gt, mask=None):
    err = np.linalg.norm(pred - gt, axis=-1)
    if mask is not None: err = err[mask]
    return np.mean(err)

def compute_bad_tau(pred, gt, tau, mask=None):
    err = np.linalg.norm(pred - gt, axis=-1)
    if mask is not None: err = err[mask]
    return np.mean(err > tau) * 100

Common pitfalls

• Evaluation is performed on images downsampled to 540×960 resolution due to memory constraints.
• Test-time optimizations such as tiling are explicitly disabled to ensure fair comparison.
• For multi-layer evaluation, single-layer models are forced to predict the same flow vector for all layers, which is an artificial workaround rather than native multi-layer prediction.

Evidence (verbatim from paper)

We adopt the commonly-used average end-point-error (EPE) and single-layer bad-$\tau$ metrics. EPE measures the average L2 distance between predicted and ground truth optical flow. Bad-$\tau$ represents the percentage of pixels having L2 error larger than a threshold of $\tau$. Evaluation is done on LayeredFlow with images downsampled to a resolution of $540\times 960$ due to memory constraints.

Citation

@misc{wen2024layeredflow,
  title={LayeredFlow: A Real-World Benchmark for Non-Lambertian Multi-Layer Optical Flow},
  author={Wen et al. (2024)},
  year={2024},
  note={arXiv:2409.05688}
}

• arXiv: 2409.05688