Emt Tracking Eval

emt-tracking-eval

EMT: A Visual Multi-Task Benchmark Dataset for Autonomous Driving — Abdel Madjid et al. (2025) (arXiv:2502.19260, 2025)

What this evaluates

Evaluates autonomous driving perception and prediction capabilities, specifically multi-agent object tracking and trajectory forecasting, using a dataset collected in the UAE with diverse driving scenarios.

Datasets

• EMT — total ?; splits: train (-1), test (-1); repo http://github.com/AV-Lab/emt-dataset

Metrics

• MOTA (primary) — range: percent
  • Multi-Object Tracking Accuracy evaluates overall tracking performance by penalizing false positives, false negatives, and identity switches relative to the number of ground truth objects.
• HOTA — range: percent
  • Higher Order Tracking Accuracy is a comprehensive metric that balances detection accuracy (DetA) and association accuracy (AssA) across different IoU thresholds.
• IDF1 — range: percent
  • Identity F1 Score measures the tracker’s ability to maintain consistent object identities by computing the F1 score of identity matches across frames.
• ADE — range: other
  • Average Displacement Error computes the mean L2 distance between predicted and ground truth trajectory points over the entire prediction horizon.
• FDE — range: other
  • Final Displacement Error computes the L2 distance between the predicted and ground truth positions at the final timestep of the prediction horizon.

Input / output format

Input: For tracking: video frames or sequences with bounding box detections (or ground truth boxes) and object class labels. For trajectory prediction: past trajectory sequences (x,y coordinates over time) of traffic agents.

Output: For tracking: predicted bounding boxes with class labels and persistent track IDs over time. For trajectory prediction: predicted future trajectory coordinates (x,y) over a specified horizon (e.g., 1s to 6s).

Scoring recipe

def compute_ade_fde(pred_traj, gt_traj):
    l2_errors = np.sqrt(np.sum((pred_traj - gt_traj)**2, axis=1))
    ade = np.mean(l2_errors)
    fde = l2_errors[-1]
    return ade, fde

def compute_mota_ids(fp, fn, ids, gt_count):
    mota = 1.0 - (fp + fn + ids) / max(gt_count, 1)
    idf1 = compute_id_f1(pred_ids, gt_ids)
    return mota, idf1

Common pitfalls

• Tracking metrics are evaluated on four superclasses (pedestrian, motorbike, cyclist, vehicle) rather than fine-grained classes, which masks performance on specific vehicle types.
• NMS threshold is lowered to 0.5 due to dense object overlap, deviating from standard tracking benchmarks and inflating false positives.
• Trajectory predictors are trained on ground truth past trajectories, isolating prediction performance from detection errors, which may overestimate real-world performance.

Evidence (verbatim from paper)

We conduct the evaluation using F1-score, Identity Switches and Higher Order Tracking Accuracy (HOTA): Multi-Object Tracking Accuracy (MOTA): Evaluates overall tracking performance by accounting for false positives, false negatives, and identity switches. Identity F1 Score (IDF1): Measures the tracker’s ability to maintain consistent object identities, considering identity switches and fragmentation. Higher Order Tracking Accuracy (HOTA): A comprehensive metric that balances detection and association accuracy, decomposed into detection accuracy (DetA) and association accuracy (AssA) components. To compare performance, we compute the average and final displacement errors for each prediction setting in a unimodal mode, as well as for the Transformer coupled with the GMM model in a multimodal output setting.

Citation

@misc{abdelmadjid2025emt,
  title={EMT: A Visual Multi-Task Benchmark Dataset for Autonomous Driving},
  author={Abdel Madjid et al. (2025)},
  year={2025},
  note={arXiv:2502.19260}
}

• arXiv: 2502.19260