Webuot 1m Eval

webuot-1m-eval

WebUOT-1M: Advancing Deep Underwater Object Tracking with A Million-Scale Benchmark — Chunhui Zhang et al. (2024) (arXiv:2405.19818, 2024)

What this evaluates

Evaluates the robustness and accuracy of deep object trackers in challenging underwater environments. It probes cross-domain adaptation from open-air to underwater domains, as well as within-domain fine-tuning capabilities, while also assessing performance under varying frame rates and complex visual conditions like occlusion and low visibility.

Datasets

• WebUOT-1M — total 1500; splits: train (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Curve of the success rate (IoU overlap) across thresholds from 0 to 1. Standard in tracking benchmarks.
• mACC — range: other
  • Mean Accuracy, calculated as the average center location error (in pixels) between predicted and ground truth bounding boxes across all frames.
• Pre — range: percent
  • Precision, the percentage of frames where the center distance between predicted and ground truth boxes is within a 20-pixel threshold.
• nPre — range: percent
  • Normalized Precision, similar to Pre but normalized by the ground truth box size to account for varying target scales.
• cAUC — range: [0, 1]
  • Complete Success Rate AUC, computed using a stricter IoU threshold range or complete overlap metric as defined in standard tracking protocols.

Input / output format

Input: Sequential video frames (underwater imagery), an initial ground-truth bounding box for the first frame, and optionally a language prompt describing the target.

Output: A sequence of predicted bounding box coordinates (x, y, width, height) for each frame in the video sequence.

Scoring recipe

def compute_metrics(predictions, ground_truths):
    ious = [iou(pred, gt) for pred, gt in zip(predictions, ground_truths)]
    auc = np.mean([np.mean([iou > t for iou in ious]) for t in np.linspace(0, 1, 101)])
    precision = np.mean([center_dist(pred, gt) <= 20 for pred, gt in zip(predictions, ground_truths)]) * 100
    macc = np.mean([center_dist(pred, gt) for pred, gt in zip(predictions, ground_truths)])
    return {'AUC': auc, 'Pre': precision, 'mACC': macc}

Common pitfalls

• Confusing Protocol I (cross-domain evaluation of pre-trained open-air trackers) with Protocol II (within-domain retraining on WebUOT-1M).
• Simulating low frame rates by randomly discarding frames can break temporal continuity, requiring careful interpolation or state reset in trackers.
• Vision-language trackers using only language prompts without bounding box cues perform significantly worse, contrary to open-air multimodal tracking trends.

Evidence (verbatim from paper)

Following*[zhang2022webuav] ; [fan2021lasot]*, we perform the one-pass evaluation (OPE) and measure trackers using five evaluation metrics (i.e., percision (Pre), normalized precision (nPre), success rate (AUC), complete success rate (cAUC), and mean accuracy (mACC)) under two protocols.

Citation

@misc{zhang2024webuot1m,
  title={WebUOT-1M: Advancing Deep Underwater Object Tracking with A Million-Scale Benchmark},
  author={Chunhui Zhang et al. (2024)},
  year={2024},
  note={arXiv:2405.19818}
}

• arXiv: 2405.19818