Huvad Eval

huvad-eval

Towards Adaptive Human-centric Video Anomaly Detection: A Comprehensive Framework and A New Benchmark — Danesh Pazho et al. (2024) (arXiv:2408.14329, 2024)

What this evaluates

Evaluates human-centric video anomaly detection models on continuously recorded real-world video streams. It measures how well models distinguish between normal and anomalous human activities across multiple camera views, particularly under continual learning conditions.

Datasets

• HuVAD — total 5196675; splits: train (4467271), test (729404); repo https://github.com/TeCSAR-UNCC/HuVAD

Metrics

• AUC-ROC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. Plots True Positive Rate against False Positive Rate across classification thresholds to measure class distinction ability.
• AUC-PR — range: [0, 1]
  • Area under the Precision-Recall curve. Calculates performance by integrating precision and recall across thresholds, better handling imbalanced datasets.
• EER — range: [0, 1]
  • Equal Error Rate. Identifies the operating threshold where False Positive Rate equals False Negative Rate, balancing sensitivity and specificity.
• 10ER — range: [0, 1]
  • 10% Error Rate. Measures the False Positive Rate at a fixed False Negative Rate of 10%, providing a practical threshold-based perspective.

Input / output format

Input: Video frames or pose sequences containing humans, with ground-truth binary labels indicating normal or anomalous activity.

Output: Continuous anomaly score per frame or video segment, used to compute threshold-dependent metrics.

Scoring recipe

import numpy as np
from sklearn.metrics import roc_auc_score, precision_recall_curve, roc_curve

def evaluate(y_true, y_scores):
    auc_roc = roc_auc_score(y_true, y_scores)
    prec, rec, _ = precision_recall_curve(y_true, y_scores)
    auc_pr = auc(rec, prec)
    fpr, tpr, _ = roc_curve(y_true, y_scores)
    eer = fpr[np.argmin(np.abs(fpr - (1 - tpr)))]
    fpr_10er = fpr[np.argmin(np.abs((1 - tpr) - 0.10))]
    return {'AUC-ROC': auc_roc, 'AUC-PR': auc_pr, 'EER': eer, '10ER': fpr_10er}

Common pitfalls

• AUC-ROC can be misleading on highly imbalanced anomaly datasets because it ignores False Negatives and may obscure key trade-offs.
• EER assumes equal costs for false positives and negatives, which may not align with real-world deployment requirements where one error type is costlier.
• 10ER fixes FNR at 10%, which requires careful threshold selection and may be too strict or lenient depending on the specific application's safety constraints.

Evidence (verbatim from paper)

AUC-ROC, AUC-PR, EER, and 10ER each provide unique insights and limitations, making their combined use essential for a comprehensive assessment. AUC-ROC measures class distinction ability by plotting the True Positive Rate (TPR) against False Positive Rate (FPR) across thresholds; however, it lacks consideration of the False Negative Rate (FNR), is sensitive to data imbalance, and may obscure key trade-offs. AUC-PR calculates the area under the Precision-Recall curve, which better handles imbalanced datasets but falls short in analyzing negative predictions and overall error distribution. EER identifies the threshold where FPR and FNR are equal, offering a balance of sensitivity (detecting anomalies) and specificity (recognizing normals), valuable in real-world deployments where the costs of false positives and negatives are balanced. Inspired by the False Match Rate 100 (FMR100) metric, this paper brings 10ER to VAD, measuring the FPR at a fixed 10% FNR—a threshold regarded as acceptable for VAD.

Citation

@misc{daneshpazho2024huvad,
  title={Towards Adaptive Human-centric Video Anomaly Detection: A Comprehensive Framework and A New Benchmark},
  author={Danesh Pazho et al. (2024)},
  year={2024},
  note={arXiv:2408.14329}
}

• arXiv: 2408.14329