Mcity Data Engine Eval

mcity-data-engine-eval

Mcity Data Engine: Iterative Model Improvement Through Open-Vocabulary Data Selection — Bogdoll et al. (2025) (arXiv:2504.21614, 2025)

What this evaluates

Evaluates an open-vocabulary data selection pipeline for iteratively improving object detection models on rare classes. Specifically, it tests whether adding newly selected and labeled frames of vulnerable road users (pedestrians, cyclists) to a seed dataset improves detection performance on fisheye traffic camera data.

Datasets

• SIP VRU Detection Dataset — total ?; splits: seed (1260), iterative (2352), validation (-1); repo https://github.com/mcity/mcity_data_engine

Metrics

• mAP@0.5 (primary) — range: [0, 1]
  • Mean Average Precision at Intersection over Union (IoU) threshold of 0.5. Computed by averaging the precision-recall curve across all confidence thresholds for the target classes (pedestrian, cyclist).
• F1 Score — range: [0, 1]
  • Harmonic mean of precision and recall calculated at a single confidence threshold, representing the balance between false positives and false negatives for deployment settings.

Input / output format

Input: Fisheye camera frames from the Smart Intersections Project traffic stream.

Output: Bounding box predictions with class labels (pedestrian, cyclist) and confidence scores. The evaluation enforces a 3/5 majority consensus across an ensemble of five open-vocabulary models before accepting a detection.

Scoring recipe

# Ensemble prediction with majority voting
preds = [model.predict(frame) for model in ensemble]
final_preds = filter_consensus(preds, min_votes=3)
# Match against ground truth
tp, fp, fn = match(final_preds, gt, iou_thresh=0.5)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall)
# mAP@0.5 across thresholds
map50 = compute_map(tp_scores, iou_thresh=0.5)

Common pitfalls

• The ensemble requires a strict 3/5 majority consensus; ignoring this leads to inflated false positive rates and invalidates the reported metrics.
• Crowd VRUs (>40 per frame) are treated as a separate task and excluded from the primary iterative dataset ($\mathcal{D}_{iter}$), so mixing them in changes the evaluation scope.
• mAP@0.5 is computed over multiple confidence thresholds, while F1/Precision/Recall are evaluated at a single deployment threshold; comparing them directly without noting the threshold difference is misleading.

Evidence (verbatim from paper)

While mAP@0.5 is the most comprehensive metric for overall model performance over multiple confidence thresholds, the remaining metrics evaluate the model performance based on a single confidence threshold, which is closer to a deployment setting. The F1 score represents the balance between recall and precision. High recall reduces false negatives, while high precision reduces false positives.

Citation

@misc{bogdoll2025mcity,
  title={Mcity Data Engine: Iterative Model Improvement Through Open-Vocabulary Data Selection},
  author={Bogdoll et al. (2025)},
  year={2025},
  note={arXiv:2504.21614}
}

• arXiv: 2504.21614