Crowdflow Eval

crowdflow-eval

Optical Flow Dataset and Benchmark for Visual Crowd Analysis — Schröder et al. (2018) (arXiv:1811.07170, 2018)

What this evaluates

Evaluates dense optical flow estimation accuracy and long-term temporal consistency in complex crowd surveillance scenarios, specifically testing robustness to non-rigid, self-occluding motion and small object tracking.

Datasets

• CrowdFlow — total 3200; splits: test (-1); repo https://github.com/tsenst/CrowdFlow

Metrics

• EPE (primary) — range: other
  • Average Endpoint Error: the mean Euclidean distance between predicted and ground-truth flow vectors across all pixels in a sequence.
• R2 — range: percent
  • Percentage of pixels where the endpoint error exceeds a tolerance threshold. The paper sets this threshold to 4 pixels (half the average body size in the dataset).
• tracking accuracy — range: percent
  • Percentage of trajectory points, advected by the estimated flow field, that lie within a specified distance threshold (15 pixels) of the corresponding ground-truth trajectory points.

Input / output format

Input: Sequential video frames capturing crowd dynamics in surveillance scenarios.

Output: Dense optical flow fields (per-pixel displacement vectors) and/or propagated trajectory coordinates.

Scoring recipe

def compute_metrics(flow_pred, flow_gt, traj_start, traj_gt, flow_field):
    # EPE
    epe = np.mean(np.sqrt(np.sum((flow_pred - flow_gt)**2, axis=-1)))
    # R2 (threshold = 4 pixels)
    r2 = np.mean((np.sqrt(np.sum((flow_pred - flow_gt)**2, axis=-1)) > 4)) * 100
    # Tracking Accuracy (threshold = 15 pixels)
    advected_pts = advect_trajectory(traj_start, flow_field) # via bilinear interpolation
    tracking_acc = np.mean(np.linalg.norm(advected_pts - traj_gt, axis=-1) <= 15) * 100
    return epe, r2, tracking_acc

Common pitfalls

• R2 uses a dataset-specific threshold of 4 pixels instead of the conventional 3 pixels.
• Long-term evaluation requires advecting trajectories through the flow field with bilinear interpolation, not just comparing adjacent frames.
• Synthetic training data often fails to generalize to real-world surveillance without domain adaptation, as noted in the benchmark's findings.

Evidence (verbatim from paper)

To assess the quality of the optical flow we propose to use two types of metrics: i) common optical flow metrics, i.e. average endpoint error (EPE) and percentage of erroneous pixel (RX) and ii) long-term motion metrics based on trajectories. Additionally, the run-time is a critical measure to assess the usability for real-time applications. For each sequence, the EPE and R2 values will be reported. While the EPE maps over the total error range, the R2 indicates the percentage of pixels with an end-point error larger than two. With R2, we set a tolerance error threshold to half of the average body size which is four pixels in our data set. To bundle the sequence results for the whole dataset the average of the sequence EPE and R2 are computed. The trajectory approach allows for a time-depending evaluation of the optical flow fields. We follow the tracking accuracy proposed in [16] for quantitative evaluations. This metric measures accumulative motion errors and disruptions from temporal inconsistencies of the flow fields. The tracking accuracy reports the percentage of tracked points from all trajectories that lie within a certain distance to the corresponding ground-truth points.

Citation

@misc{schroder2018crowdflow,
  title={Optical Flow Dataset and Benchmark for Visual Crowd Analysis},
  author={Schröder et al. (2018)},
  year={2018},
  note={arXiv:1811.07170}
}

• arXiv: 1811.07170