Tb Bench Eval

tb-bench-eval

TB-Bench: Training and Testing Multi-Modal AI for Understanding Spatio-Temporal Traffic Behaviors from Dashcam Images/Videos — Charoenpitaks et al. (2025) (arXiv:2501.05733, 2025)

What this evaluates

This benchmark evaluates multi-modal large language models' ability to understand spatio-temporal traffic behaviors from ego-centric dashcam images and videos. It probes eight distinct perception tasks, including road detection, object-lane alignment, turning prediction, and ego-trajectory estimation, requiring both spatial reasoning and temporal tracking.

Datasets

• TB-Bench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted answers across all instances. Calculated as (number of correct predictions / total number of predictions) * 100. Reported per task and averaged across all eight tasks.

Input / output format

Input: A sequence of N dashcam frames (N=8 for temporal tasks, N=1 otherwise), each resized to 384×384 pixels and processed into 16 visual embeddings per frame, paired with a text prompt containing a question and, for classification tasks, a list of multiple-choice options.

Output: A text string containing either the selected class label from the provided options or a numerical value formatted as specified in the prompt (e.g., 'xx.x meters').

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        pred_clean = pred.strip().lower()
        gold_clean = gold.strip().lower()
        if pred_clean == gold_clean or gold_clean in pred_clean:
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• In-context learning uses different numbers of examples depending on the task type: 3 examples for single-frame tasks vs. 1 example for multi-frame tasks.
• Numerical output questions require strict adherence to the prompt's formatting instruction (e.g., 'Answer in xx.x meters') to avoid parsing failures during evaluation.
• The random guess baseline is reported as 0.0% accuracy rather than the theoretical chance level, which can skew comparative analysis if not accounted for.

Evidence (verbatim from paper)

Table 4: Results of compared methods on TB-Bench are reported in accuracy (%), where higher indicates better performance. Random guess† results are considered zero. ⋆In-context learning for single-frame tasks uses three in-context examples, while multi-frame tasks use one.

Citation

@misc{charoenpitaks2025tbbench,
  title={TB-Bench: Training and Testing Multi-Modal AI for Understanding Spatio-Temporal Traffic Behaviors from Dashcam Images/Videos},
  author={Charoenpitaks et al. (2025)},
  year={2025},
  note={arXiv:2501.05733}
}

• arXiv: 2501.05733