Tad Eval

tad-eval

TAD: A Large-Scale Benchmark for Traffic Accidents Detection from Video Surveillance — Xu et al. (2022) (arXiv:2209.12386, 2022)

What this evaluates

Evaluates computer vision models on detecting traffic accidents from surveillance footage across image classification, video classification, and object detection tasks. It probes the model's ability to distinguish accident scenarios from normal traffic and localize accident events in real-world highway scenes.

Datasets

• TAD — total 24810; splits: train (-1), val (-1), test (1490); repo https://github.com/yajunbaby/TAD-benchmark

Metrics

• F1-score (primary) — range: [0, 1]
  • Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall). Used as the headline metric for binary accident classification tasks.
• mAP@0.5 — range: [0, 1]
  • Mean Average Precision at Intersection over Union (IoU) threshold of 0.5. Averages the AP across all four accident categories (roll over, wreck, collision, victims).
• Recall — range: [0, 1]
  • True Positive Rate: TP / (TP + FN). Measures the proportion of actual accidents correctly identified.
• Precision — range: [0, 1]
  • Positive Predictive Value: TP / (TP + FP). Measures the proportion of predicted accidents that are correct.
• AP — range: [0, 1]
  • Average Precision across all confidence thresholds for a single class.
• AR — range: [0, 1]
  • Average Recall across all confidence thresholds and IoU thresholds for a single class.

Input / output format

Input: RGB images or video clips (16 frames, 112x112 pixels) captured from third-person surveillance cameras on highways.

Output: Binary class label (Accident/Normal) for classification tasks; bounding boxes with class labels (roll over, wreck, collision, victims) and confidence scores for object detection.

Scoring recipe

def compute_metrics(preds, gold):
    # Classification
    tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    
    # Object Detection (per class)
    aps = []
    for cls in classes:
        cls_preds = get_preds_for_class(preds, cls)
        cls_gold = get_gold_for_class(gold, cls)
        aps.append(calculate_ap(cls_preds, cls_gold, iou_thresh=0.5))
    map_05 = sum(aps) / len(aps)
    return f1, map_05, precision, recall

Common pitfalls

• Bounding boxes are expanded by 1/3 from the minimum external rectangle to capture background features, which inflates detection metrics compared to standard tight annotations.
• The test set is fixed across all three tasks (1490 images / same video source), so performance gains may not be independent across image classification, video classification, and object detection.
• A fixed confidence threshold of 0.4 is used to convert object detection outputs into binary accident predictions, which is non-standard for mAP calculation and affects Recall/Precision/F1.

Evidence (verbatim from paper)

TAD randomly extracted 24,810 labeled images from 333 videos, 4 main accident types. ... TAD is split into train, validation and test sets. ... Evaluation metrics for objection detection are AR, AP and MAP(0.5) while results for tests on single image are measured by Recall, Precision and F1-score to present comparison with image classification. The comparison results of image classification task within the three datasets evaluated by Recall, Precision and F1-score in Table 3.

Citation

@misc{xu2022tad,
  title={TAD: A Large-Scale Benchmark for Traffic Accidents Detection from Video Surveillance},
  author={Xu et al. (2022)},
  year={2022},
  note={arXiv:2209.12386}
}

• arXiv: 2209.12386