Trek 150 Eval

trek-150-eval

Visual Object Tracking in First Person Vision — Dunnhofer et al. (2022) (arXiv:2209.13502, 2022)

What this evaluates

Evaluates single-object visual tracking performance in first-person vision videos, specifically testing robustness to object manipulation, occlusions, and dynamic interactions under real-time execution constraints.

Datasets

• TREK-150 — total 150; splits: test (-1)

Metrics

• SS, NPS, GSR (primary) — range: [0, 1]
  • Standard VOT metrics: Success Score (SS) based on IoU threshold, Normalized Precision Score (NPS) measuring center location error, and Generalized Success Rate (GSR). Computed per the VOT2017 protocol.
• FPS — range: other
  • Frames per second, measuring processing speed/efficiency.

Input / output format

Input: Video sequence of first-person vision frames with a ground-truth bounding box provided only for the first frame.

Output: A sequence of bounding boxes (one per executed frame) tracking the target object throughout the video.

Scoring recipe

def evaluate_rte(tracker, video_frames, fps):
    predictions = []
    t = 0
    while t < len(video_frames):
        box = tracker.run(video_frames[t])
        predictions.append(box)
        exec_time = tracker.get_execution_time()
        t += int(exec_time * fps)
    ss = compute_success_score(predictions, ground_truth)
    nps = compute_normalized_precision(predictions, ground_truth)
    gsr = compute_generalized_success_rate(predictions, ground_truth)
    return ss, nps, gsr

Common pitfalls

• Assuming standard One-Pot Evaluation (OPE) where the tracker processes every frame sequentially; RTE skips frames during tracker execution.
• Ignoring the real-time constraint: trackers must report processing time per frame to correctly align predictions with the actual video timeline.
• Using generic tracking metrics without accounting for FPV-specific challenges like rapid appearance changes due to object manipulation.

Evidence (verbatim from paper)

The overall performance scores (SS, NPS, GSR) are ultimately obtained as for the OPE protocol. Together with those values, we evaluate the trackers’ processing speed in frames per second (FPS) to quantify their efficiency.

Citation

@misc{dunnhofer2022visual,
  title={Visual Object Tracking in First Person Vision},
  author={Dunnhofer et al. (2022)},
  year={2022},
  note={arXiv:2209.13502}
}

• arXiv: 2209.13502