Numosim Eval

numosim-eval

NUMOSIM: A Synthetic Mobility Dataset with Anomaly Detection Benchmarks — Stanford et al. (2024) (arXiv:2409.03024, 2024)

What this evaluates

Evaluates geospatial anomaly detection models on synthetic human mobility data. It probes the ability of algorithms to identify injected anomalous movement patterns across different granularities (staypoint, trip, agent) while controlling for demographic, temporal, and spatial factors.

Datasets

• NUMOSIM — total ?; splits: train (-1), test (-1)

Metrics

• Average Precision (AP) (primary) — range: [0, 1]
  • Area under the precision-recall curve, computed by averaging precision at each recall threshold. Ideal for highly imbalanced anomaly detection where anomalies are rare.
• AUCROC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds.

Input / output format

Input: Labeled mobility sequences containing staypoints, trips, and agent trajectories with demographic, geospatial, and temporal features. Test instances contain injected anomalies alongside normal patterns.

Output: Continuous anomaly scores per instance (staypoint, trip, agent, or point), optionally aggregated to agent-level via strategies like max-pooling.

Scoring recipe

def compute_metrics(scores, labels):
    # scores: predicted anomaly scores (higher = more anomalous)
    # labels: ground truth binary flags (1=anomalous, 0=normal)
    precision, recall, _ = precision_recall_curve(labels, scores)
    ap = auc(recall, precision)
    fpr, tpr, _ = roc_curve(labels, scores)
    aucroc = auc(fpr, tpr)
    return {"AP": ap, "AUCROC": aucroc}

Common pitfalls

• Anomaly prevalence rate is adjustable by selecting specific test subsets, drastically affecting AP/AUCROC scores.
• Models are trained on anomaly-free data but evaluated on data with injected anomalies, requiring unsupervised/semi-supervised adaptation.
• Evaluation granularity varies (staypoint, trip, agent, point); failing to aggregate scores correctly (e.g., max-pooling) causes level mismatches.
• Simple statistical baselines (e.g., POI visit rate) often outperform complex deep learning models, contrary to typical ML expectations.

Evidence (verbatim from paper)

Finally, we evaluate each model using metrics such as the Average Precision (AP) and Area Under the Receiver Operating Characteristic curve (AUCROC). Although other metrics like Maximum F1-Score or Average Precision Recall are viable, they not are included in this analysis.

Citation

@misc{stanford2024numosim,
  title={NUMOSIM: A Synthetic Mobility Dataset with Anomaly Detection Benchmarks},
  author={Stanford et al. (2024)},
  year={2024},
  note={arXiv:2409.03024}
}

• arXiv: 2409.03024