Lamot Eval

lamot-eval

LaMOT: Language-Guided Multi-Object Tracking — Li et al. (2024) (arXiv:2406.08324, 2024)

What this evaluates

Evaluates open-vocabulary multi-object tracking by requiring models to follow multiple targets in video sequences guided by natural language descriptions. It probes the model's ability to jointly perform text-grounded detection and long-term identity association across diverse, real-world scenarios.

Datasets

• LaMOT — total 1660; splits: full (1660); repo https://github.com/Nathan-Li123/LaMOT

Metrics

• HOTA (primary) — range: [0, 100] percent
  • Higher Order Tracking Accuracy, a composite metric that balances detection accuracy (DetA) and association accuracy (AssA) across IoU thresholds.
• MOTA — range: [0, 100] percent
  • Multiple Object Tracking Accuracy, measuring tracking performance based on false negatives, false positives, and identity switches.
• IDF1 — range: [0, 100] percent
  • ID F1 score, the harmonic mean of ID precision and ID recall, evaluating identity consistency over time.

Input / output format

Input: Video frames with corresponding natural language descriptions specifying the target objects to track.

Output: Time-series bounding boxes with assigned track IDs, matched to the provided language queries.

Scoring recipe

def evaluate_mot(preds, gold):
    tp, fp, fn = match_boxes(preds, gold, iou_thresh=0.5)
    ids = count_identity_switches(preds, gold)
    mota = 1 - (fn + fp + ids) / max(len(gold), 1)
    hota = compute_hota(tp, fp, fn, ids)
    idf1 = harmonic_mean(id_precision(preds, gold), id_recall(preds, gold))
    return {'HOTA': hota, 'MOTA': mota, 'IDF1': idf1}

Common pitfalls

• Feature-based ReID yields minimal gains because language descriptions make visually distinct targets appear semantically similar.
• Models fail on open-vocabulary queries if not explicitly designed for text-grounded detection, as they cannot recognize unseen categories.
• Scenario difficulty varies drastically; drone footage is significantly harder due to high target density and low resolution, skewing aggregate scores if not analyzed per scenario.

Evidence (verbatim from paper)

As in Tab.[3], LaMOTer achieves the best performance. For instance, it achieves 48.45% in HOTA, and 47.66% in IDF1.

Citation

@misc{li2024lamot,
  title={LaMOT: Language-Guided Multi-Object Tracking},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2406.08324}
}

• arXiv: 2406.08324