Roundabout Tau Eval

roundabout-tau-eval

TAU-R1: Visual Language Model for Traffic Anomaly Understanding — Lin et al. (2026) (arXiv:2603.19098, 2026)

What this evaluates

This benchmark evaluates a model's ability to detect traffic anomalies in roadside surveillance videos and generate detailed, reasoning-grounded textual summaries of those anomalies. It probes both binary/fine-grained classification accuracy and semantic alignment of generated descriptions with ground-truth event narratives.

Datasets

• Roundabout-TAU — total 342; splits: train (300), test (42); repo https://github.com/siri-rouser/TAU-R1

Metrics

• 4-cls AP (primary) — range: [0, 1]
  • Average Precision across four traffic anomaly classes, computed from predicted class probabilities against ground-truth labels.
• 4-cls F1 — range: [0, 1]
  • Macro-averaged F1 score across four traffic anomaly classes.
• 2-cls AP — range: [0, 1]
  • Average Precision for binary anomaly detection (anomalous vs. normal).
• 2-cls F1 — range: [0, 1]
  • F1 score for binary anomaly detection.
• BLEU — range: [0, 1]
  • Standard n-gram overlap metric for text generation, typically BLEU-4.
• METEOR — range: [0, 1]
  • Metric evaluating alignment between generated and reference text using synonymy, stemming, and exact matches.
• ROUGE-L — range: [0, 1]
  • Recall-Oriented Understudy for Gisting Evaluation based on longest common subsequence.
• G-Score — range: [0, 10]
  • LLM-judged score (0–10) summing four aspects: environment correctness (0–1), object grounding (0–2), description quality (0–5), and reasoning quality (0–2).

Input / output format

Input: Video clips (frames/timestamps) for classification; video clips paired with anomaly event prompts/questions for summarization.

Output: Classification: predicted class label (4-class or binary). Summarization: free-text description/summary of the anomalous event.

Scoring recipe

def score_classification(preds, gold):
    ap_4cls = compute_average_precision(preds['4cls'], gold['4cls'])
    f1_4cls = compute_f1(preds['4cls'], gold['4cls'])
    ap_2cls = compute_average_precision(preds['2cls'], gold['2cls'])
    f1_2cls = compute_f1(preds['2cls'], gold['2cls'])
    return ap_4cls, f1_4cls, ap_2cls, f1_2cls

def score_summarization(pred_text, ref_text):
    bleu = compute_bleu(pred_text, ref_text)
    meteor = compute_meteor(pred_text, ref_text)
    rouge_l = compute_rouge_l(pred_text, ref_text)
    aspects = llm_judge_score(pred_text, ref_text, prompt_template)
    g_score = aspects.env_correctness + aspects.object_grounding + aspects.desc_quality + aspects.reasoning_quality
    return bleu, meteor, rouge_l, g_score

Common pitfalls

• Summarization metrics are only computed on the 34 anomalous test videos, not all 42 test videos.
• G-Score relies on an external LLM judge (ChatGPT-5) with specific prompt templates provided only in supplementary material, making exact replication difficult.
• Test set is small (42 videos) and class distribution is maintained from the full dataset, which may lead to high variance in AP/F1 scores.

Evidence (verbatim from paper)

We evaluate TAU-R1 on the two tasks defined in Roundabout-TAU: anomaly classification and anomaly summarization. For classification, we report both four-class and binary classification performance using Average Precision (AP) and F1 score. For summarization, following prior video anomaly understanding work [zhang2025holmes, xing2025echotraffic], we use standard text-generation metrics including BLEU [papineni2002bleu], METEOR [banerjee2005meteor] and ROUGE-L [lin2004rouge]. In addition to those traditional text-based method, we further introduce GPT-Eval to better assess semantic accuracy and reasoning quality, following recent VLM evaluation practice [liu2023visual, tang2024hawk]. Specifically, we prompt ChatGPT-5 to score each prediction from 0 to 10 from four aspects: environment correctness (0–1), object grounding (0–2), description quality (0–5), and reasoning quality (0–2).

Citation

@misc{lin2026tau_r1,
  title={TAU-R1: Visual Language Model for Traffic Anomaly Understanding},
  author={Lin et al. (2026)},
  year={2026},
  note={arXiv:2603.19098}
}

• arXiv: 2603.19098