R U Maad Eval

r-u-maad-eval

A Benchmark for Unsupervised Anomaly Detection in Multi-Agent Trajectories — Wiederer et al. (2022) (arXiv:2209.01838, 2022)

What this evaluates

Evaluates a model's ability to perform unsupervised anomaly detection on multi-agent traffic trajectories in urban environments. It probes frame-wise recognition of rare and abnormal driving behaviors, including both individual maneuvers and context-dependent interactions between agents and static map features.

Datasets

• R-U-MAAD — total 245574; splits: train (205942), val (39472), test (160); repo https://github.com/againerju/r_u_maad

Metrics

• frame-wise anomaly detection accuracy (primary) — range: [0, 1]
  • Binary classification accuracy computed per time step (normal vs abnormal). Ignore regions corresponding to transition phases are explicitly excluded from the calculation, ensuring errors in those steps do not affect the final score.

Input / output format

Input: Multi-agent trajectory sequences represented as 2D birds-eye view center coordinates sampled at 10 Hz. Models receive 2 seconds of historical trajectory data to evaluate or predict the next 3 seconds.

Output: Per time step: binary anomaly label (normal or abnormal) or anomaly score. Predictions must align with the frame-wise annotation timeline and exclude ignore regions.

Scoring recipe

def compute_frame_wise_accuracy(predictions, gold_labels, ignore_mask):
    valid_preds = [p for p, ign in zip(predictions, ignore_mask) if not ign]
    valid_gold = [g for g, ign in zip(gold_labels, ignore_mask) if not ign]
    if len(valid_gold) == 0:
        return 0.0
    correct = sum(1 for p, g in zip(valid_preds, valid_gold) if p == g)
    return correct / len(valid_gold)

Common pitfalls

• Including 'ignore regions' (transition phases) in the evaluation, which the protocol explicitly excludes from the calculation.
• Assuming the test set is purely simulated; it is a hybrid where only one agent is animated in simulation while others are replayed from real Argoverse data.
• Treating all 13 anomaly classes uniformly without considering the actor-interactive vs. map-interactive distinctions, which significantly impacts algorithm design and failure analysis.

Evidence (verbatim from paper)

We define abnormal driving as the set of all uncommon and rare scenarios and provide a test set with frame-wise annotations of diverse driving anomalies... Ignore regions are not considered during evaluation, such that errors in ignored time steps do not contribute to the result.

Citation

@misc{wiederer2022rumaad,
  title={A Benchmark for Unsupervised Anomaly Detection in Multi-Agent Trajectories},
  author={Wiederer et al. (2022)},
  year={2022},
  note={arXiv:2209.01838}
}

• arXiv: 2209.01838