Webuav 3m Eval

webuav-3m-eval

WebUAV-3M: A Benchmark for Unveiling the Power of Million-Scale Deep UAV Tracking — Zhang et al. (2022) (arXiv:2201.07425, 2022)

What this evaluates

Evaluates the robustness and accuracy of deep visual tracking algorithms on large-scale, real-world UAV video sequences. It probes how well trackers handle diverse environmental conditions, motion dynamics, and target appearance changes without parameter tuning.

Datasets

• WebUAV-3M — total 3300000; splits: test (-1); repo https://github.com/983632847/WebUAV-3M

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the success plot curve. It calculates the percentage of frames where the overlap score (IoU) between predicted and ground-truth bounding boxes exceeds a given threshold, averaged over thresholds from 0 to 1.
• Pre — range: [0, 1]
  • Precision plot score. Measures the percentage of frames whose center location error is within a predefined threshold (e.g., 20 pixels).
• nPre — range: [0, 1]
  • Normalized precision plot. Normalizes the precision score over the size of the ground-truth bounding box to reduce sensitivity to target size and image resolution.
• mAcc — range: [0, 1]
  • Mean accuracy. Encourages trackers to output invisible predictions when the target disappears, measuring robustness to target loss.
• cAUC — range: [0, 1]
  • Complete success plot. Area under the curve of a complete overlap score $S_c = \frac{|B_G \cap B_P|}{|B_G \cup B_P|} - \frac{d^2(b^G, b^P)}{c^2} - \alpha v$, which combines IoU, normalized center distance, and aspect ratio consistency.

Input / output format

Input: Video frames and an initial ground-truth bounding box for the target.

Output: Predicted bounding box coordinates (x, y, w, h) for each frame in the sequence.

Scoring recipe

def compute_auc(pred_boxes, gt_boxes):
    ious = [intersection_over_union(p, g) for p, g in zip(pred_boxes, gt_boxes)]
    thresholds = np.linspace(0, 1, 101)
    success_rates = [sum(iou >= t for iou in ious) / len(ious) for t in thresholds]
    return np.trapz(success_rates, thresholds)

def compute_pre(pred_boxes, gt_boxes, threshold=20):
    errors = [center_distance(p, g) for p, g in zip(pred_boxes, gt_boxes)]
    return sum(e <= threshold for e in errors) / len(errors)

Common pitfalls

• Evaluation is strictly one-pass (OPE) with no re-initialization, providing a lower-bound performance estimate compared to multi-pass protocols.
• Standard metrics (Pre, AUC) ignore aspect ratio changes; use cAUC for complete geometric evaluation.
• No hyperparameter tuning is performed on the benchmark; all trackers use default weights and fixed parameters.

Evidence (verbatim from paper)

In this work, we perform a one-pass evaluation (OPE) and adopt four popular metrics, i.e., the precision plot (Pre), normalized precision plot (nPre), success plot (AUC) and mean accuracy (mAcc) measures, and a newly proposed metric, the complete success plot (cAUC), to assess the performance of different tracking algorithms.

Citation

@misc{zhang2022webuav3m,
  title={WebUAV-3M: A Benchmark for Unveiling the Power of Million-Scale Deep UAV Tracking},
  author={Zhang et al. (2022)},
  year={2022},
  note={arXiv:2201.07425}
}

• arXiv: 2201.07425