Road Anomaly Seg Eval

road-anomaly-seg-eval

Latency-aware Road Anomaly Segmentation in Videos: A Photorealistic Dataset and New Metrics — Tian et al. (2024) (arXiv:2401.04942, 2024)

What this evaluates

Evaluates video-level road anomaly segmentation models in autonomous driving scenarios, specifically probing their ability to maintain prediction validity over time sequences and perform under real-time latency constraints.

Datasets

• Road Anomaly Segmentation Dataset — total 220; splits: train (200), val (20)

Metrics

• latency-aware metrics (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic (AUROC) and False Positive Rate at 95% True Positive Rate (FPR95) computed over video sequences, adjusted to account for inference latency constraints during streaming evaluation.
• temporal consistency — range: [0, 1]
  • Measures the stability of anomaly predictions across consecutive frames in a video sequence, penalizing flickering or inconsistent segmentations over time.

Input / output format

Input: Video sequences of road scenes (synthetic CARLA data) with ground-truth anomaly segmentation masks.

Output: Per-frame anomaly segmentation masks for each video sequence.

Scoring recipe

def evaluate(predictions, gold, timestamps, latency_budget):
    valid_preds = [p for p, t in zip(predictions, timestamps) if t <= latency_budget]
    valid_gold = [g for g, t in zip(gold, timestamps) if t <= latency_budget]
    frame_scores = [compute_iou(p, g) for p, g in zip(valid_preds, valid_gold)]
    auroc = compute_auroc(frame_scores, valid_gold)
    fpr95 = compute_fpr_at_tpr95(frame_scores, valid_gold)
    consistency = compute_frame_to_frame_iu_correlation(predictions)
    return auroc, fpr95, consistency

Common pitfalls

• Evaluating only on latency-agnostic metrics ignores real-world streaming constraints, overestimating practical utility.
• Retraining models on extra anomalous data often improves latency-agnostic scores but degrades temporal consistency due to overfitting to specific anomaly appearances.
• High latency-agnostic performance correlates with higher sensitivity to latency drops, making latency-aware evaluation essential for safety-critical deployment.

Evidence (verbatim from paper)

During evaluation, we utilize an additional set of scenes without anomalous objects for the training of the semantic segmentation task, which is a preceding task of anomaly segmentation. Of the 220 video sequences, 200 are used for training (if needed) and 20 are used for validation. The results evaluated on the proposed metrics, i.e., the latency-agnostic, latency-aware metrics as well as the temporal consistency, are reported in Table 1.

Citation

@misc{tian2024latency,
  title={Latency-aware Road Anomaly Segmentation in Videos: A Photorealistic Dataset and New Metrics},
  author={Tian et al. (2024)},
  year={2024},
  note={arXiv:2401.04942}
}

• arXiv: 2401.04942