Mot16 Eval

mot16-eval

MOT16: A Benchmark for Multi-Object Tracking — Milan et al. (2016) (arXiv:1603.00831, 2016)

What this evaluates

Evaluates multi-object tracking algorithms on video sequences by measuring detection accuracy, identity consistency, and localization precision. It assesses how well trackers maintain object identities over time while correctly handling occlusions, distractors, and varying crowd densities.

Datasets

• MOT16 — total ?; splits: test (-1)

Metrics

• MOTA (primary) — range: percent
  • 1 - (sum(FN + FP + IDSW) / sum(GT)) over all frames, where FN=missed targets, FP=false alarms, IDSW=identity switches, GT=ground truth objects.
• MOTP — range: percent
  • sum(d_{t,i}) / sum(c_t) over all frames and matches, where d is bounding box overlap dissimilarity and c_t is number of matches in frame t.
• IDSW — range: count
  • Count of identity switches where a ground truth target is matched to a different track ID than its previous temporal assignment.
• FM — range: count
  • Count of track fragmentations where a trajectory is interrupted and later resumed.
• MT/ML — range: ratio
  • Ratio of mostly tracked (≥80% lifespan) or mostly lost (<20% lifespan) trajectories to total ground truth trajectories.

Input / output format

Input: Per-frame bounding box detections (x, y, width, height, confidence) and optional class labels.

Output: Per-frame bounding boxes with unique trajectory IDs, confidence scores, and class labels.

Scoring recipe

# Initialize counters
sum_gt = sum_fn = sum_fp = sum_idsw = 0
sum_overlap = sum_matches = 0
# Process all test sequences concatenated
for frame in sequences:
    matches = hungarian(gt_boxes, pred_boxes, dist_thresh=0.5)
    sum_gt += len(gt_boxes)
    sum_matches += len(matches)
    for gt, pred in matches:
        sum_overlap += 1 - iou(gt, pred)
    sum_fn += len(gt_boxes) - len(matches)
    sum_fp += len(pred_boxes) - len(matches)
    # Track IDSW & FM via temporal matching logic...
sum_gt = max(sum_gt, 1)
mota = 1 - (sum_fn + sum_fp + sum_idsw) / sum_gt
motp = sum_overlap / sum_matches if sum_matches > 0 else 0

Common pitfalls

• MOTA alone does not measure localization accuracy; MOTP must be reported alongside it.
• Averaging metrics per sequence is discouraged; the benchmark requires concatenating all test sequences before computing totals.
• Distractors, static people, and reflections are explicitly excluded from evaluation (neither penalized nor rewarded).
• ID switches are counted based on temporal matching constraints, not just single-frame overlap.

Evidence (verbatim from paper)

The MOTA [41] is perhaps the most widely used metric to evaluate a tracker's performance. The main reason for this is its expressiveness as it combines three sources of errors defined above: $$ \mathrm {M O T A} = 1 - \frac {\sum_ {t} \left(\mathrm {F N} _ {t} + \mathrm {F P} _ {t} + \mathrm {I D S W} _ {t}\right)}{\sum_ {t} \mathrm {G T} _ {t}}, \tag{1} $$ where $t$ is the frame index and GT is the number of ground truth objects. We report the percentage MOTA $(-\infty, 100]$ in our benchmark.

Citation

@misc{milan2016mot16,
  title={MOT16: A Benchmark for Multi-Object Tracking},
  author={Milan et al. (2016)},
  year={2016},
  note={arXiv:1603.00831}
}

• arXiv: 1603.00831