Uvh 26 Eval

uvh-26-eval

The Urban Vision Hackathon Dataset and Models: Towards Image Annotations and Accurate Vision Models for Indian Traffic — Akash Sharma et al. (2025) (arXiv:2511.02563, 2025)

What this evaluates

Evaluates object detection models on a domain-specific Indian traffic dataset, probing their ability to localize and classify 14 heterogeneous vehicle types under surveillance viewpoints with varying occlusion and scale.

Datasets

• UVH-26 — total 26646; splits: test (400)

Metrics

• mAP(50:95) (primary) — range: [0, 1]
  • Mean of Average Precision (AP) values computed at Intersection over Union (IoU) thresholds from 0.50 to 0.95 in steps of 0.05.
• mAP(75) — range: [0, 1]
  • AP computed at a strict IoU threshold of 0.75, emphasizing precise localization quality.
• mAP(50) — range: [0, 1]
  • AP computed at a lenient IoU threshold of 0.50, reflecting coarse detection capability.

Input / output format

Input: High-resolution traffic images captured from surveillance/CCTV viewpoints.

Output: Bounding boxes with class labels for 14 India-specific vehicle categories.

Scoring recipe

def compute_map(predictions, ground_truth, iou_thresholds):
    aps = []
    for iou in iou_thresholds:
        ap = 0.0
        for class_id in classes:
            # Standard COCO-style AP: match predictions to GT using IoU >= iou,
            # compute precision-recall curve, integrate area under curve
            ap += calculate_ap(predictions[class_id], ground_truth[class_id], iou)
        aps.append(ap / len(classes))
    return sum(aps) / len(aps)

# mAP(50:95) = compute_map(preds, gts, np.arange(0.50, 0.96, 0.05))

Common pitfalls

• COCO and UVH-26 annotate 2-Wheelers/Cycles differently (COCO excludes riders, UVH-26 includes them), so these classes were explicitly excluded from cross-dataset comparisons.
• The 'Others' umbrella class was excluded from evaluation due to low instance counts and lack of semantic specificity.
• Domain shift between ego-view (COCO) and top-down surveillance (UVH-26) significantly impacts baseline generalization, requiring careful fine-tuning rather than direct transfer.

Evidence (verbatim from paper)

We use a held-out test set curated from our gold dataset comprising of 400 images, sampled to ensure diverse coverage of all fourteen UVH-26 vehicle classes. ... Performance assessment follows standard practices widely adopted in the object detection literature. The primary metric is the mean Average Precision (mAP), evaluated across a range of Intersection over Union (IoU) thresholds. In particular, we report: 1. mAP(50:95): The main benchmark metric, defined as the mean of AP values at IoU thresholds from 0.50 to 0.95 in steps of 0.05. 2. mAP(75): AP computed at a stricter IoU threshold of 0.75, which emphasizes precise localization quality. 3. mAP(50): AP computed at a lenient IoU threshold of 0.50, reflecting the model’s capacity for coarse but correct detections.

Citation

@misc{sharma2025uvh,
  title={The Urban Vision Hackathon Dataset and Models: Towards Image Annotations and Accurate Vision Models for Indian Traffic},
  author={Akash Sharma et al. (2025)},
  year={2025},
  note={arXiv:2511.02563}
}

• arXiv: 2511.02563