Vad Anticipation Eval

vad-anticipation-eval

A New Comprehensive Benchmark for Semi-supervised Video Anomaly Detection and Anticipation — Cao et al. (2023) (arXiv:2305.13611, 2023)

What this evaluates

Evaluates a model's ability to detect anomalous events in surveillance videos and anticipate their occurrence in future frames. It specifically probes scene-dependent anomaly recognition and multi-step temporal anticipation.

Datasets

• ShanghaiTech — total ?; splits: test (-1)
• CUHK Avenue — total ?; splits: test (-1)
• IITB Corridor — total ?; splits: test (-1)
• NWPU Campus — total ?; splits: test (-1)
• ShanghaiTech-sd — total ?; splits: test (-1)

Metrics

• AUC (%) (primary) — range: percent
  • Area under the Receiver Operating Characteristic (ROC) curve computed over concatenated frame-level anomaly scores across the entire dataset.

Input / output format

Input: 256×256 pixel crops centered on objects detected by ByteTrack, processed in sequences of 8 frames.

Output: Frame-level anomaly scores for detection (current frame) and anticipation (future frames up to 3 seconds ahead).

Scoring recipe

# 1. Compute frame-level anomaly scores (e.g., reconstruction error)
frame_scores = model.predict(video_crops)
# 2. Concatenate all frame scores across the dataset
all_scores = np.concatenate(frame_scores)
# 3. Compute ROC AUC against ground truth frame labels
auc_percent = roc_auc_score(ground_truth_labels, all_scores) * 100

Common pitfalls

• Computing AUC per video instead of concatenating all frames across the dataset first.
• Failing to account for scene-dependent anomalies (e.g., treating 'cycling' as abnormal in all scenes rather than only specific ones).
• Using inaccurate object tracking on low-resolution datasets (like CUHK Avenue), which degrades crop quality and scores.

Evidence (verbatim from paper)

We use the area under the curve (AUC) of receiver operating characteristic (ROC) to evaluate the performance for both VAD and VAA. Note that we concatenate all the frames in a dataset and then compute the overall frame-level AUC, which is widely adopted.

Citation

@misc{cao2023nwpu,
  title={A New Comprehensive Benchmark for Semi-supervised Video Anomaly Detection and Anticipation},
  author={Cao et al. (2023)},
  year={2023},
  note={arXiv:2305.13611}
}

• arXiv: 2305.13611